Augmented reality head-mounted display with eye tracking for pupil steering

ABSTRACT

A head-mounted display device for providing augmented reality contents to a wearer includes a light projector, and eye tracker, and a beam steerer. The light projector is configured to project light for rendering images based at least on the augmented reality contents. The eye tracker is configured to determine a position of a pupil of an eye of the wearer. The beam steerer is configured to change a direction of the light from the light projector based on the position of the pupil.

RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 16/222,789, entitled “Integrated Augmented RealityHead-Mounted Display for Pupil Steering” filed Dec. 17, 2018, whichclaims the benefit of, and the priority to, U.S. Provisional PatentApplication Ser. No. 62/599,793, entitled “Integrated Augmented RealityHead-Mounted Display for Pupil Steering” filed Dec. 18, 2017, both ofwhich are incorporated by reference herein in their entireties. Thisapplication is related to U.S. patent application Ser. No. ______,entitled “Augmented Reality Head-Mounted Display with a Fresnel Combinerand Pupil Steering” filed concurrently herewith (Attorney Docket Number010235-01-5186-US), U.S. patent application Ser. No. ______, entitled“Augmented Reality Head-Mounted Display with a Pancake Combiner andPupil Steering” filed concurrently herewith (Attorney Docket Number010235-01-5187-US), U.S. patent application Ser. No. ______, entitled“Augmented Reality Head-Mounted Display with Beam Shifter for PupilSteering” filed concurrently herewith (Attorney Docket Number010235-01-5189-US), U.S. patent application Ser. No. ______, entitled“Augmented Reality Head-Mounted Display with a Focus-SupportingProjector for Pupil Steering” filed concurrently herewith (AttorneyDocket Number 010235-01-5190-US), and U.S. patent application Ser. No.______, entitled “Eye Tracking for Pupil Steering in Head-MountedDisplays Using Eye Tracking Sensors” filed concurrently herewith(Attorney Docket Number 010235-01-5201-US). All of these applicationsare incorporated by reference herein in their entireties.

TECHNICAL FIELD

This relates generally to display devices, and more specifically tohead-mounted display devices.

BACKGROUND

Head-mounted display devices (also called herein head-mounted displays)are gaining popularity as means for providing visual information to auser. For example, the head-mounted display devices are used for virtualreality and augmented reality operations.

However, the size and weight of conventional head-mounted displays havelimited applications of head-mounted displays.

SUMMARY

Accordingly, there is a need for head-mounted displays that are compactand light, thereby enhancing the user's virtual-reality and/or augmentedreality experience.

In particular, conventional head-mounted display devices (e.g.,conventional head-mounted display devices configured for augmentedreality operations) project images over a large area around an eye of auser in order to provide a wide field of view in all gaze-directions(e.g., in order to deal with pupil steering). However, projecting imagesover a large area leads to reduced brightness of the projected images.Compensating for the reduced brightness typically requires a highintensity light source, which is typically large and heavy, and has highpower consumption.

The above deficiencies and other problems associated with conventionalhead-mounted displays are reduced or eliminated by the disclosed displaydevices. In accordance with some embodiments, a position of a pupil ofan eye of a user is determined (e.g., using an eye tracker) and imagesare projected over a reduced area toward the pupil of the eye. Byreducing the area over which the images are projected, the need for ahigh intensity light source is reduced or eliminated. This, in turn,allows compact, light, and low power-consumption head-mounted displays.

In accordance with some embodiments, a head-mounted display device forproviding augmented reality contents to a wearer includes an eyetracker, a light projector, a beam steerer and a combiner. The eyetracker is configured to determine a position of a pupil of an eye ofthe wearer, the light projector is configured to project light forrendering images based at least on the augmented reality contents, thebeam steerer is configured to change a direction of the light from thelight projector based on the position of the pupil, and the combiner isconfigured to combine the light from the light projector and light froman outside of the head-mounted display device for providing an overlapof the rendered image and a real image that corresponds to the lightfrom the outside of the head-mounted display device.

In accordance with some embodiments, a method providing augmentedreality contents to a wearer using a head-mounted display device thatincludes an eye tracker, a light projector, a beam steerer, and acombiner includes determining a position of a pupil of an eye with theeye tracker. The method also includes projecting, with the lightprojector, light for rendering images based at least on the augmentedreality contents and changing, with the beam steerer, a direction of thelight from the light projector based on the position of the pupil. Themethod further includes combining, with the combiner, the light from thelight projector and light from an outside of the head-mounted displaydevice for providing an overlap of the rendered image and a real imagethat corresponds to the light from the outside of the head-mounteddisplay device.

In accordance with some embodiments, a method for providing images to awearer using a head-mounted display device including a light projectorand a beam shifter includes projecting, with the light projector, lightfor rendering images based at least on virtual reality contents and/oraugmented reality contents and changing, with a beam shifter, a path ofthe light projected from the light projector based on a position of apupil of an eye of the wearer.

In accordance with some embodiments, a head-mounted display device forproviding images to a wearer includes a light projector configured toproject light for rendering images based at least on virtual realitycontents and/or augmented reality contents and a beam shifter configuredto change a path of the light projected from the light projector basedon a position of a pupil of an eye of the wearer.

In accordance with some embodiments, a head-mounted display device forproviding augmented reality contents to a wearer includes a first lightprojector configured to project light for rendering images based atleast on the augmented reality contents, and a first Fresnel combinerconfigured to combine the light from the first light projector and lightfrom an outside of the head-mounted display device for providing anoverlap of the rendered image and a real image that corresponds to thelight from the outside of the head-mounted display device.

In accordance with some embodiments, a method providing augmentedreality contents to a wearer using a head-mounted display device thatincludes a first light projector and a first Fresnel combiner includesprojecting, with the first light projector, light for rendering an imagebased at least on the augmented reality contents and combining, with thefirst Fresnel combiner, the light from the first light projector andlight from an outside of the head-mounted display device for providingan overlap of the rendered image and a real image that corresponds tothe light from the outside of the head-mounted display device.

In accordance with some embodiments, a head-mounted display device forproviding augmented reality contents to a wearer includes a lightprojector and a pancake combiner. The light projector is configured toproject a light having a first polarization for rendering images basedat least on the augmented reality contents. The pancake combiner isconfigured to combine the light from the light projector and light froman outside of the head-mounted display device for providing an overlapof the rendered image and a real image that corresponds to the lightfrom the outside of the head-mounted display device. The pancakecombiner is also configured to direct the light from the light projectortoward a pupil of an eye the wearer.

In accordance with some embodiments, a method providing augmentedreality contents to a wearer using a head-mounted display deviceincluding a light projector and a pancake combiner includes projecting,with the light projector, a light having a first polarization forrendering an image based at least on the augmented reality contents andcombining, with the pancake combiner, the light from the light projectorand light from an outside of the head-mounted display device forproviding an overlap of the rendered image and a real image thatcorresponds to the light from the outside of the head-mounted displaydevice. The pancake combiner is configured to direct the light from thelight projector toward a pupil of an eye the wearer.

In accordance with some embodiments, a head-mounted display device forproviding augmented reality contents to a wearer includes a lightprojector configured to project light for rendering images based atleast on the augmented reality contents, an eye tracker configured todetermine a position of a pupil of an eye of the wearer, and a beamsteerer configured to change a direction of the light from the lightprojector based on the position of the pupil.

In accordance with some embodiments, a method for providing augmentedreality contents to a wearer using a head-mounted display device thatincludes an eye tracker, a light projector, and a beam steerer includesdetermining, with the eye tracker, a position of a pupil of an eye ofthe wearer and projecting, with the light projector, light for renderingimages based at least on the augmented reality contents. The method alsoincludes changing, with the beam steerer, a direction of the light fromthe light projector based on the position of the pupil.

In accordance with some embodiments, a method for providing augmentedreality contents to a wearer using a head-mounted display device thatincludes an eye tracking sensor, a light projector, a beam steerer, anda combiner, includes determining, with the eye tracking sensor, aposition of a pupil of an eye of the wearer and projecting, with thelight projector, light for rendering images based at least on theaugmented reality contents. The method also includes changing, with thebeam steerer, a direction of the light from the light projector based onthe position of the pupil. The light from the beam steerer is directedtoward the combiner, and the light from the beam steerer and light froman outside of the head-mounted display device are combined, by thecombiner, to provide an overlap of a rendered image and a real imagethat corresponds to the light from the outside of the head-mounteddisplay device.

In accordance with some embodiments, a head-mounted display device forproviding augmented reality contents to a wearer, the device includes aneye tracking sensor configured to determine a position of a pupil of aneye of the wearer, a light projector configured to project light forrendering images based at least on the augmented reality contents, abeam steerer configured to change a direction of the light from thelight projector based on the position of the pupil, and a combinerconfigured to combine the light from the light projector and light froman outside of the head-mounted display device for providing an overlapof the rendered image and a real image that corresponds to the lightfrom the outside of the head-mounted display device.

In accordance with some embodiments, a head-mounted display device forproviding images to a wearer includes a focus-supporting light projectorconfigured to project light for rendering images. The light projectedfrom the focus-supporting light projector corresponds to an image planethat is selected based at least in part on a position of a pupil of aneye of the wearer. The device also includes a beam steerer configured tochange a path of the light projected from the focus-supporting lightprojector based on the position of the pupil of the eye of the wearer.

In accordance with some embodiments, a method for providing images to awearer is performed using a head-mounted display device that includes afocus-supporting light projector and a beam steerer. The method includesprojecting, with the focus-supporting light projector, light forrendering images based at least on virtual reality contents and/oraugmented reality contents. The light projected from thefocus-supporting light projector corresponds to an image plane that isselected based at least in part on a position of a pupil of an eye ofthe wearer. The method also includes changing, with the beam steerer, apath of the light projected from the focus-supporting light projectorbased on the position of the pupil of the eye of the wearer.

Thus, the disclosed embodiments provide compact and light displaydevices that can be used for augmented reality and/or virtual realityoperations.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1 is a perspective view of a display device in accordance with someembodiments.

FIG. 2 is a block diagram of a system including a display device inaccordance with some embodiments.

FIG. 3 is an isometric view of a display device in accordance with someembodiments.

FIG. 4A is a schematic diagram illustrating a display device inaccordance with some embodiments.

FIG. 4B is a schematic diagram illustrating a display device inaccordance with some embodiments.

FIG. 4C is a schematic diagram illustrating a display device inaccordance with some embodiments.

FIG. 4D is a schematic diagram illustrating a display device inaccordance with some embodiments.

FIGS. 5A-5B are schematic diagrams illustrating example operations of adisplay device in accordance with some embodiments.

FIGS. 5C-5E are schematic diagrams illustrating a display device inaccordance with some embodiments.

FIG. 5F is a schematic diagram illustrating a scanning reflectorassembly in accordance with some embodiments.

FIG. 5G-5H are schematic diagrams illustrating a display device inaccordance with some embodiments.

FIG. 5I is a schematic diagram illustrating changing of direction ofprojected light in accordance with some embodiments.

FIGS. 6A-6C are schematic diagrams illustrating a tunable waveguide inaccordance with some embodiments.

FIG. 7A is a schematic diagram illustrating a holographic combiner inaccordance with some embodiments.

FIG. 7B is a schematic diagram illustrating a holographic combiner inaccordance with some embodiments.

FIG. 8 is a schematic diagram illustrating a aspheric combiner inaccordance with some embodiments.

FIG. 9A is a schematic diagram illustrating a Fresnel combiner inaccordance with some embodiments.

FIG. 9B is a schematic diagram illustrating a Fresnel combiner inaccordance with some embodiments.

FIG. 9C is a schematic diagram illustrating a Fresnel combiner inaccordance with some embodiments.

FIG. 9D is a schematic diagram illustrating a Fresnel combiner inaccordance with some embodiments.

FIG. 9E is a schematic diagram illustrating a Fresnel combiner inaccordance with some embodiments.

FIGS. 10A and 10B are schematic diagrams illustrating a pancake combinerin accordance with some embodiments.

FIG. 11A is a schematic diagram illustrating detecting a position of apupil by imaging in accordance with some embodiments.

FIG. 11B is a schematic diagram illustrating detecting a position of apupil by glint tracking in accordance with some embodiments.

FIG. 11C is a schematic diagrams illustrating four structured patternsof light used for eye tracking in accordance with some embodiments.

FIG. 11D is a schematic diagrams illustrating detecting a position of apupil by infrared (IR) retinal reflex detection in accordance with someembodiments.

FIG. 11E is a schematic diagrams illustrating detecting a position of apupil by depth measurement in accordance with some embodiments.

FIG. 11F is a schematic diagrams illustrating detecting a position of apupil by depth scanning in accordance with some embodiments.

FIGS. 11G-11I are schematic diagrams illustrating detecting a positionof a pupil by a time-of-flight detector in accordance with someembodiments.

FIG. 11J is a schematic diagram illustrating detecting a position of apupil by a time-of-flight detector in accordance with some embodiments.

FIGS. 12A-12E are schematic diagrams illustrating head-mounted displayswith focus-supporting light projectors in accordance with someembodiments.

FIGS. 12F and 12G are schematic diagrams illustrating example operationsof a head-mounted display in accordance with some embodiments.

These figures are not drawn to scale unless indicated otherwise.

DETAILED DESCRIPTION

Utilizing optical elements (e.g., combiners) that transmit real-worldimages and reflect computer-generated images allows augmented realityoperations without requiring a separate high-resolution high frame ratecamera, thereby reducing the size and the weight of head-mounteddisplays. In order to provide images to a pupil regardless of a movementof the pupil, conventional combiners project images onto a large eyebox(e.g., an eyebox having a characteristic dimension, such as a diameteror a width, of at least 1 cm). A large eyebox can be achieved, forexample, by a pupil replication technique that expands the size of abeam transmitted to the eyebox. For example, conventional combiners mayinclude a waveguide coupled with an optical element (e.g., a grating) toexpand the size of the beam. However, when light is projected onto alarge eyebox, a significant portion of the light lands on an areaoutside the pupil and, hence, is not detected. This leads to decreasedbrightness of the projected images. Instead of increasing the power ofdisplays, which increases the size, weight, and power consumption ofhead-mounted displays, images are projected onto a small eyebox (e.g.,an eyebox that corresponds to a size of the pupil), thereby improvingthe brightness of the projected images. To accommodate for the movementof the pupil and reduce vignetting of the projected light, a position ofthe pupil is determined using an eye tracker and the projected light issteered toward the pupil.

Reference will now be made to embodiments, examples of which areillustrated in the accompanying drawings. In the following description,numerous specific details are set forth in order to provide anunderstanding of the various described embodiments. However, it will beapparent to one of ordinary skill in the art that the various describedembodiments may be practiced without these specific details. In otherinstances, well-known methods, procedures, components, circuits, andnetworks have not been described in detail so as not to unnecessarilyobscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are used onlyto distinguish one element from another. For example, a first lightprojector could be termed a second light projector, and, similarly, asecond light projector could be termed a first light projector, withoutdeparting from the scope of the various described embodiments. The firstlight projector and the second light projector are both lightprojectors, but they are not the same light projector.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof. The term “exemplary” is used herein in the senseof “serving as an example, instance, or illustration” and not in thesense of “representing the best of its kind.”

FIG. 1 illustrates display device 100 in accordance with someembodiments. In some embodiments, display device 100 is configured to beworn on a head of a user (e.g., by having the form of spectacles oreyeglasses, as shown in FIG. 1) or to be included as part of a helmetthat is to be worn by the user. When display device 100 is configured tobe worn on a head of a user or to be included as part of a helmet,display device 100 is called a head-mounted display. Alternatively,display device 100 is configured for placement in proximity of an eye oreyes of the user at a fixed location, without being head-mounted (e.g.,display device 100 is mounted in a vehicle, such as a car or anairplane, for placement in front of an eye or eyes of the user). Asshown in FIG. 1, display device 100 includes display 110. Display 110 isconfigured for presenting visual contents (e.g., augmented realitycontents, virtual reality contents, mixed reality contents, or anycombination thereof) to a user.

In some embodiments, display device 100 includes one or more componentsdescribed herein with respect to FIG. 2. In some embodiments, displaydevice 100 includes additional components not shown in FIG. 2.

FIG. 2 is a block diagram of system 200 in accordance with someembodiments. The system 200 shown in FIG. 2 includes display device 205(which corresponds to display device 100 shown in FIG. 1), imagingdevice 235, and input interface 240 that are each coupled to console210. While FIG. 2 shows an example of system 200 including one displaydevice 205, imaging device 235, and input interface 240, in otherembodiments, any number of these components may be included in system200. For example, there may be multiple display devices 205 each havingassociated input interface 240 and being monitored by one or moreimaging devices 235, with each display device 205, input interface 240,and imaging devices 235 communicating with console 210. In alternativeconfigurations, different and/or additional components may be includedin system 200. For example, in some embodiments, console 210 isconnected via a network (e.g., the Internet) to system 200 or isself-contained as part of display device 205 (e.g., physically locatedinside display device 205). In some embodiments, display device 205 isused to create mixed reality by adding in a view of the realsurroundings. Thus, display device 205 and system 200 described here candeliver augmented reality, virtual reality, and mixed reality.

In some embodiments, as shown in FIG. 1, display device 205 is ahead-mounted display that presents media to a user. Examples of mediapresented by display device 205 include one or more images, video,audio, or some combination thereof. In some embodiments, audio ispresented via an external device (e.g., speakers and/or headphones) thatreceives audio information from display device 205, console 210, orboth, and presents audio data based on the audio information. In someembodiments, display device 205 immerses a user in an augmentedenvironment.

In some embodiments, display device 205 also acts as an augmentedreality (AR) headset. In these embodiments, display device 205 augmentsviews of a physical, real-world environment with computer-generatedelements (e.g., images, video, sound, etc.). Moreover, in someembodiments, display device 205 is able to cycle between different typesof operation. Thus, display device 205 operate as a virtual reality (VR)device, an augmented reality (AR) device, as glasses or some combinationthereof (e.g., glasses with no optical correction, glasses opticallycorrected for the user, sunglasses, or some combination thereof) basedon instructions from application engine 255.

Display device 205 includes electronic display 215, one or moreprocessors 216, eye tracking module 217, adjustment module 218, one ormore locators 220, one or more position sensors 225, one or moreposition cameras 222, memory 228, inertial measurement unit (IMU) 230,one or more reflective elements 260 or a subset or superset thereof(e.g., display device 205 with electronic display 215, one or moreprocessors 216, and memory 228, without any other listed components).Some embodiments of display device 205 have different modules than thosedescribed here. Similarly, the functions can be distributed among themodules in a different manner than is described here.

One or more processors 216 (e.g., processing units or cores) executeinstructions stored in memory 228. Memory 228 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR RAM or other random access solidstate memory devices; and may include non-volatile memory, such as oneor more magnetic disk storage devices, optical disk storage devices,flash memory devices, or other non-volatile solid state storage devices.Memory 228, or alternately the non-volatile memory device(s) withinmemory 228, includes a non-transitory computer readable storage medium.In some embodiments, memory 228 or the computer readable storage mediumof memory 228 stores programs, modules and data structures, and/orinstructions for displaying one or more images on electronic display215.

Electronic display 215 displays images to the user in accordance withdata received from console 210 and/or processor(s) 216. In variousembodiments, electronic display 215 may comprise a single adjustabledisplay element or multiple adjustable display elements (e.g., a displayfor each eye of a user). In some embodiments, electronic display 215 isconfigured to display images to the user by projecting the images ontoone or more reflective elements 260.

In some embodiments, the display element includes one or more lightemission devices and a corresponding array of spatial light modulators.A spatial light modulator is an array of electro-optic pixels,opto-electronic pixels, some other array of devices that dynamicallyadjust the amount of light transmitted by each device, or somecombination thereof. These pixels are placed behind one or more lenses.In some embodiments, the spatial light modulator is an array of liquidcrystal based pixels in an LCD (a Liquid Crystal Display). Examples ofthe light emission devices include: an organic light emitting diode, anactive-matrix organic light-emitting diode, a light emitting diode, sometype of device capable of being placed in a flexible display, or somecombination thereof. The light emission devices include devices that arecapable of generating visible light (e.g., red, green, blue, etc.) usedfor image generation. The spatial light modulator is configured toselectively attenuate individual light emission devices, groups of lightemission devices, or some combination thereof. Alternatively, when thelight emission devices are configured to selectively attenuateindividual emission devices and/or groups of light emission devices, thedisplay element includes an array of such light emission devices withouta separate emission intensity array. In some embodiments, electronicdisplay 215 projects images to one or more reflective elements 260,which reflect at least a portion of the light toward an eye of a user.

One or more lenses direct light from the arrays of light emissiondevices (optionally through the emission intensity arrays) to locationswithin each eyebox and ultimately to the back of the user's retina(s).An eyebox is a region that is occupied by an eye of a user locatedproximity to display device 205 (e.g., a user wearing display device205) for viewing images from display device 205. In some cases, theeyebox is represented as a 10 mm×10 mm square. In some embodiments, theone or more lenses include one or more coatings, such as anti-reflectivecoatings.

In some embodiments, the display element includes an infrared (IR)detector array that detects IR light that is retro-reflected from theretinas of a viewing user, from the surface of the corneas, lenses ofthe eyes, or some combination thereof. The IR detector array includes anIR sensor or a plurality of IR sensors that each correspond to adifferent position of a pupil of the viewing user's eye. In alternateembodiments, other eye tracking systems may also be employed.

Eye tracking module 217 determines locations of each pupil of a user'seyes. In some embodiments, eye tracking module 217 instructs electronicdisplay 215 to illuminate the eyebox with IR light (e.g., via IRemission devices in the display element).

A portion of the emitted IR light will pass through the viewing user'spupil and be retro-reflected from the retina toward the IR detectorarray, which is used for determining the location of the pupil.Alternatively, the reflection off of the surfaces of the eye is used toalso determine location of the pupil. The IR detector array scans forretro-reflection and identifies which IR emission devices are activewhen retro-reflection is detected. Eye tracking module 217 may use atracking lookup table and the identified IR emission devices todetermine the pupil locations for each eye. The tracking lookup tablemaps received signals on the IR detector array to locations(corresponding to pupil locations) in each eyebox. In some embodiments,the tracking lookup table is generated via a calibration procedure(e.g., user looks at various known reference points in an image and eyetracking module 217 maps the locations of the user's pupil while lookingat the reference points to corresponding signals received on the IRtracking array). As mentioned above, in some embodiments, system 200 mayuse other eye tracking systems than the embedded IR one describedherein.

Adjustment module 218 generates an image frame based on the determinedlocations of the pupils. In some embodiments, this sends a discreteimage to the display that will tile subimages together thus a coherentstitched image will appear on the back of the retina. Adjustment module218 adjusts an output (i.e. the generated image frame) of electronicdisplay 215 based on the detected locations of the pupils. Adjustmentmodule 218 instructs portions of electronic display 215 to pass imagelight to the determined locations of the pupils. In some embodiments,adjustment module 218 also instructs the electronic display to not passimage light to positions other than the determined locations of thepupils. Adjustment module 218 may, for example, block and/or stop lightemission devices whose image light falls outside of the determined pupillocations, allow other light emission devices to emit image light thatfalls within the determined pupil locations, translate and/or rotate oneor more display elements, dynamically adjust curvature and/or refractivepower of one or more active lenses in the lens (e.g., microlens) arrays,or some combination thereof.

Optional locators 220 are objects located in specific positions ondisplay device 205 relative to one another and relative to a specificreference point on display device 205. A locator 220 may be a lightemitting diode (LED), a corner cube reflector, a reflective marker, atype of light source that contrasts with an environment in which displaydevice 205 operates, or some combination thereof. In embodiments wherelocators 220 are active (i.e., an LED or other type of light emittingdevice), locators 220 may emit light in the visible band (e.g., about400 nm to 750 nm), in the infrared band (e.g., about 750 nm to 1 mm), inthe ultraviolet band (about 100 nm to 400 nm), some other portion of theelectromagnetic spectrum, or some combination thereof.

In some embodiments, locators 220 are located beneath an outer surfaceof display device 205, which is transparent to the wavelengths of lightemitted or reflected by locators 220 or is thin enough to notsubstantially attenuate the wavelengths of light emitted or reflected bylocators 220. Additionally, in some embodiments, the outer surface orother portions of display device 205 are opaque in the visible band ofwavelengths of light. Thus, locators 220 may emit light in the IR bandunder an outer surface that is transparent in the IR band but opaque inthe visible band.

IMU 230 is an electronic device that generates calibration data based onmeasurement signals received from one or more position sensors 225.Position sensor 225 generates one or more measurement signals inresponse to motion of display device 205. Examples of position sensors225 include: one or more accelerometers, one or more gyroscopes, one ormore magnetometers, another suitable type of sensor that detects motion,a type of sensor used for error correction of IMU 230, or somecombination thereof. Position sensors 225 may be located external to IMU230, internal to IMU 230, or some combination thereof.

Based on the one or more measurement signals from one or more positionsensors 225, IMU 230 generates first calibration data indicating anestimated position of display device 205 relative to an initial positionof display device 205. For example, position sensors 225 includemultiple accelerometers to measure translational motion (forward/back,up/down, left/right) and multiple gyroscopes to measure rotationalmotion (e.g., pitch, yaw, roll). In some embodiments, IMU 230 rapidlysamples the measurement signals and calculates the estimated position ofdisplay device 205 from the sampled data. For example, IMU 230integrates the measurement signals received from the accelerometers overtime to estimate a velocity vector and integrates the velocity vectorover time to determine an estimated position of a reference point ondisplay device 205. Alternatively, IMU 230 provides the sampledmeasurement signals to console 210, which determines the firstcalibration data. The reference point is a point that may be used todescribe the position of display device 205. While the reference pointmay generally be defined as a point in space; however, in practice thereference point is defined as a point within display device 205 (e.g., acenter of IMU 230).

In some embodiments, IMU 230 receives one or more calibration parametersfrom console 210. As further discussed below, the one or morecalibration parameters are used to maintain tracking of display device205. Based on a received calibration parameter, IMU 230 may adjust oneor more IMU parameters (e.g., sample rate). In some embodiments, certaincalibration parameters cause IMU 230 to update an initial position ofthe reference point so it corresponds to a next calibrated position ofthe reference point. Updating the initial position of the referencepoint as the next calibrated position of the reference point helpsreduce accumulated error associated with the determined estimatedposition. The accumulated error, also referred to as drift error, causesthe estimated position of the reference point to “drift” away from theactual position of the reference point over time.

Imaging device 235 generates calibration data in accordance withcalibration parameters received from console 210. Calibration dataincludes one or more images showing observed positions of locators 220that are detectable by imaging device 235. In some embodiments, imagingdevice 235 includes one or more still cameras, one or more videocameras, any other device capable of capturing images including one ormore locators 220, or some combination thereof. Additionally, imagingdevice 235 may include one or more filters (e.g., used to increasesignal to noise ratio). Imaging device 235 is configured to optionallydetect light emitted or reflected from locators 220 in a field of viewof imaging device 235. In embodiments where locators 220 include passiveelements (e.g., a retroreflector), imaging device 235 may include alight source that illuminates some or all of locators 220, whichretro-reflect the light towards the light source in imaging device 235.Second calibration data is communicated from imaging device 235 toconsole 210, and imaging device 235 receives one or more calibrationparameters from console 210 to adjust one or more imaging parameters(e.g., focal length, focus, frame rate, ISO, sensor temperature, shutterspeed, aperture, etc.).

In some embodiments, display device 205 optionally includes one or morereflective elements 260. In some embodiments, electronic display device205 optionally includes a single reflective element 260 or multiplereflective elements 260 (e.g., a reflective element 260 for each eye ofa user). In some embodiments, electronic display device 215 projectscomputer-generated images on one or more reflective elements 260, which,in turn, reflect the images toward an eye or eyes of a user. Thecomputer-generated images include still images, animated images, and/ora combination thereof. The computer-generated images include objectsthat appear to be two-dimensional and/or three-dimensional objects. Insome embodiments, one or more reflective elements 260 are partiallytransparent (e.g., the one or more reflective elements 260 have atransmittance of at least 15%, 20%, 25%, 30%, 35%, 40%, 45%, or 50%),which allows transmission of ambient light. In such embodiments,computer-generated images projected by electronic display 215 aresuperimposed with the transmitted ambient light (e.g., transmittedambient image) to provide augmented reality images.

Input interface 240 is a device that allows a user to send actionrequests to console 210. An action request is a request to perform aparticular action. For example, an action request may be to start or endan application or to perform a particular action within the application.Input interface 240 may include one or more input devices. Example inputdevices include: a keyboard, a mouse, a game controller, data from brainsignals, data from other parts of the human body, or any other suitabledevice for receiving action requests and communicating the receivedaction requests to console 210. An action request received by inputinterface 240 is communicated to console 210, which performs an actioncorresponding to the action request. In some embodiments, inputinterface 240 may provide haptic feedback to the user in accordance withinstructions received from console 210. For example, haptic feedback isprovided when an action request is received, or console 210 communicatesinstructions to input interface 240 causing input interface 240 togenerate haptic feedback when console 210 performs an action.

Console 210 provides media to display device 205 for presentation to theuser in accordance with information received from one or more of:imaging device 235, display device 205, and input interface 240. In theexample shown in FIG. 2, console 210 includes application store 245,tracking module 250, and application engine 255. Some embodiments ofconsole 210 have different modules than those described in conjunctionwith FIG. 2. Similarly, the functions further described herein may bedistributed among components of console 210 in a different manner thanis described here.

When application store 245 is included in console 210, application store245 stores one or more applications for execution by console 210. Anapplication is a group of instructions, that when executed by aprocessor, is used for generating content for presentation to the user.Content generated by the processor based on an application may be inresponse to inputs received from the user via movement of display device205 or input interface 240. Examples of applications include: gamingapplications, conferencing applications, video playback application, orother suitable applications.

When tracking module 250 is included in console 210, tracking module 250calibrates system 200 using one or more calibration parameters and mayadjust one or more calibration parameters to reduce error indetermination of the position of display device 205. For example,tracking module 250 adjusts the focus of imaging device 235 to obtain amore accurate position for observed locators on display device 205.Moreover, calibration performed by tracking module 250 also accounts forinformation received from IMU 230. Additionally, if tracking of displaydevice 205 is lost (e.g., imaging device 235 loses line of sight of atleast a threshold number of locators 220), tracking module 250re-calibrates some or all of system 200.

In some embodiments, tracking module 250 tracks movements of displaydevice 205 using second calibration data from imaging device 235. Forexample, tracking module 250 determines positions of a reference pointof display device 205 using observed locators from the secondcalibration data and a model of display device 205. In some embodiments,tracking module 250 also determines positions of a reference point ofdisplay device 205 using position information from the first calibrationdata. Additionally, in some embodiments, tracking module 250 may useportions of the first calibration data, the second calibration data, orsome combination thereof, to predict a future location of display device205. Tracking module 250 provides the estimated or predicted futureposition of display device 205 to application engine 255.

Application engine 255 executes applications within system 200 andreceives position information, acceleration information, velocityinformation, predicted future positions, or some combination thereof ofdisplay device 205 from tracking module 250. Based on the receivedinformation, application engine 255 determines content to provide todisplay device 205 for presentation to the user. For example, if thereceived information indicates that the user has looked to the left,application engine 255 generates content for display device 205 thatmirrors the user's movement in an augmented environment. Additionally,application engine 255 performs an action within an applicationexecuting on console 210 in response to an action request received frominput interface 240 and provides feedback to the user that the actionwas performed. The provided feedback may be visual or audible feedbackvia display device 205 or haptic feedback via input interface 240.

FIG. 3 is an isometric view of display device 300 in accordance withsome embodiments. In some other embodiments, display device 300 is partof some other electronic display (e.g., a digital microscope, ahead-mounted display device, etc.). In some embodiments, display device300 includes light emission device array 310 and one or more lenses 330.In some embodiments, display device 300 also includes an IR detectorarray.

Light emission device array 310 emits image light and optional IR lighttoward the viewing user. Light emission device array 310 may be, e.g.,an array of LEDs, an array of microLEDs, an array of OLEDs, or somecombination thereof. Light emission device array 310 includes lightemission devices 320 that emit light in the visible light (andoptionally includes devices that emit light in the IR).

In some embodiments, display device 300 includes an emission intensityarray configured to selectively attenuate light emitted from lightemission array 310. In some embodiments, the emission intensity array iscomposed of a plurality of liquid crystal cells or pixels, groups oflight emission devices, or some combination thereof. Each of the liquidcrystal cells is, or in some embodiments, groups of liquid crystal cellsare, addressable to have specific levels of attenuation. For example, ata given time, some of the liquid crystal cells may be set to noattenuation, while other liquid crystal cells may be set to maximumattenuation. In this manner, the emission intensity array is able tocontrol what portion of the image light emitted from light emissiondevice array 310 is passed to the one or more lenses 330. In someembodiments, display device 300 uses an emission intensity array tofacilitate providing image light to a location of pupil 350 of eye 340of a user, and minimize the amount of image light provided to otherareas in the eyebox.

One or more lenses 330 receive the modified image light (e.g.,attenuated light) from emission intensity array (or directly fromemission device array 310), and direct the modified image light to alocation of pupil 350.

An optional IR detector array detects IR light that has beenretro-reflected from the retina of eye 340, a cornea of eye 340, acrystalline lens of eye 340, or some combination thereof. The IRdetector array includes either a single IR sensor or a plurality of IRsensitive detectors (e.g., photodiodes). In some embodiments, the IRdetector array is separate from light emission device array 310. In someembodiments, the IR detector array is integrated into light emissiondevice array 310.

In some embodiments, light emission device array 310 and an emissionintensity array make up a display element. Alternatively, the displayelement includes light emission device array 310 (e.g., when lightemission device array 310 includes individually adjustable pixels)without the emission intensity array. In some embodiments, the displayelement additionally includes the IR array. In some embodiments, inresponse to a determined location of pupil 350, the display elementadjusts the emitted image light such that the light output by thedisplay element is refracted by one or more lenses 330 toward thedetermined location of pupil 350, and not toward other locations in theeyebox.

In some embodiments, display device 300 includes one or more broadbandsources (e.g., one or more white LEDs) coupled with a plurality of colorfilters, in addition to, or instead of, light emission device array 310.

FIG. 4A is a schematic diagram illustrating display device 400 inaccordance with some embodiments. In some embodiments, display device400 corresponds to display device 100 described herein with respect toFIG. 1. In some embodiments, display device 400 is configured to provideaugmented reality contents to a wearer of display device 400. In FIG.4A, display device 400 includes light projector 402, beam steerer 404,combiner 410, and eye tracker 408. Light projector 402 projects light414-1 toward beam steerer 404, which, in turn, directs light 414-1toward beam combiner 410. Beam combiner 410 reflects and/or guides atleast a portion of light 414-1 toward pupil 406 (e.g., a pupil of an eyeof a user or wearer of display device 400). Beam combiner 410 combineslight 414-1 with light (e.g., light 416) coming from the outside ofdisplay device 400 (e.g., ambient light) so that an image represented bylight 414-1 is overlapped with, or superimposed on, a real-world imageprovided by light 416.

Eye tracker 408 is configured to determine a position of pupil 406and/or track its movement as pupil 406 rotates toward different gazedirections. In some embodiments, eye tracker 408 corresponds to, iscoupled with, or is included in eye tracking module 217 described hereinwith respect to FIG. 2. In some embodiments, determining a position ofpupil 406 includes determining the position of pupil 406 on an x-y planeof pupil 406 (e.g., reference plane 407-1). In some embodiments, the x-yplane is a curvilinear plane. In some embodiments, determining aposition of pupil 406 includes determining a distance between the eyeand eye tracker 408 (e.g., the shortest distance between the eye and eyetracker 408). In some embodiments, eye tracker 408 includes a lightsource (e.g., an infrared or a near-infrared light source). In someembodiments, eye tracker 408 is integrated with light projector 402. Insome embodiments, light projected by light projector 402 and lightdetected by eye tracker (e.g., IR light) 408 have the same optical path(or parallel optical paths) and are transmitted or guided by the sameoptical elements (e.g., one or more lenses 412, beam steerer 404 and/orbeam combiner 410).

In some embodiments, light projector 402 is configured to project lightfor providing augmenter reality images overlapped with real-world view.In some embodiments, light projector 402 includes one or more lightemission devices. Examples of the light emission devices include: lightemitting diodes (LEDs), superluminescent light emitting diodes (SLEDs),lasers, or some combination thereof. The light emission devices includedevices that are capable of generating visible light (e.g., red, green,blue, etc.) used for multi-color image generation. In some embodiments,the light emission devices also include devices that generate infrared(IR) and/or near infrared (NIR) light. In some embodiments, the one ormore light emission devices includes a liquid crystal display (LCD), aliquid crystal on silicon (LCOS) display, organic light emitting diodes(OLEDs), inorganic light emitting diodes (ILEDs), digital lightprocessing (DLP) display, or any combination thereof. In someembodiments, the one or more light emission devices are opticallycoupled with a corresponding array of spatial light modulators. Thearray of spatial light modulators is configured to selectively attenuateindividual light emission devices, groups of light emission devices, orany combination thereof. In FIG. 4A, pupil 406 is located adjacent toreference plane 407-1 that is parallel to an x-y plane and tangential toa surface of pupil 406. Light 414-1 projected by light projector 402illuminates an area on reference plane 407-1. In some embodiments, thearea is sized to cover only a subset of an eye box configured to coverall possible positions of a pupil. For example, the area covers an areathat has a characteristic dimension less than 20 mm (e.g., a round areahaving a diameter of 7 mm, 6 mm, 5 mm, 4 mm, or 3 mm, a square area of 7mm×7 mm, 6 mm×6 mm, 5 mm×5 mm, 4 mm×4 mm, or 3 mm×3 mm, etc.). In someembodiments, light projector 402 projects light over an area onreference plane 407-1 having a characteristic dimension of at least 3mm. In some embodiments, light 414-1 is steered to project light withinan area of the reference plane 407-1 plane, which intersects with anuncertainty cone encompassing positions of pupil 406 directed to allpossible gaze directions.

Beam steerer 404 directs light 414-1 toward combiner 410, which, inturn, guides light 414-1 toward pupil 406. In some embodiments, beamsteerer 404 adjusts the direction of light 414-1 and/or offsets light414-1 based on the position of pupil 406 determined by eye tracker 408.In some embodiments, beam steerer 404 includes a mechanical beam steererincluding one or more actuators that change the location of beam steerer404 in the x- and/or y-directions, or in the x-, y- and/or z-directionswith respect to display device 400. In some embodiments, beam steerer404 includes a mechanical beam steerer including one or more actuatorsconfigured to rotate about one or more axes. In some embodiments, beamsteerer 404 includes one or more translational, one or more rotationalmirrors, or any combination thereof. In some embodiments, beam steerer404 also includes one or more stationary mirrors. In some embodiments,beam steerer 404 is integrated with light projector 402 or beam combiner410.

Combiner 410 reflects and/or guides light 414-1 projected by lightprojector 402 toward pupil 406 and transmits light 416 from the outsideof display device 400. As a result, computer-generated images formed bylight projected from light projector 402 are overlapped with areal-world image. In some embodiments, combiner 410 is configured toavoid pupil replication. For example, combiner 410 reflects or guideslight (e.g., light 414-1) onto an area without replicated rays. In someembodiments, combiner 410 includes a Fresnel combiner, a pancakecombiner, an ellipsoidal mirror, one or more tunable waveguides, or aholographic combiner.

Optionally, display device 400 includes one or more lenses 412. In someembodiments, one or more lenses 412 are optically coupled with lightprojector 402 and positioned on the optical path of light 414-1 beforebeam steerer 404. In some embodiments, lenses 412 are optically coupledwith beam steerer 404 and positioned on the optical path of light 414-1after beam steerer 404. In some embodiments, one or more lenses 412focus light 414-1 projected by light projector 402. In some embodiments,one or more lenses 412 include a lens selected from a group consistingof a concave lens, a convex lens, a plano-concave lens, a plano-convexlens, or a convex-concave lens. In some embodiments, one or more lenses412 include a lens selected from a group consisting of a spherical lensor an aspherical lens. In some embodiments, one or more lenses 412include a Fresnel lens including one or two Fresnel surfaces, at least aportion of a Fresnel surface being defined by a plurality of Fresnelstructures. In some embodiments, one or more lenses 412 include anadaptive lens with an adjustable focal distance (e.g., an autofocusinglens, an electro-wetting lens, a liquid lens, or a liquid crystal lens).

FIG. 4B is a schematic diagram illustrating display device 420 inaccordance with some embodiments. Display device 420 corresponds todisplay device 400 described herein with respect to FIG. 4A except that,in FIG. 4B, light projector 402 is integrated with beam steerer 404.Component 422 projects and steers light 414-1 toward combiner 410,which, in turn, reflects light 414-1 toward pupil 406.

FIG. 4C is a schematic diagram illustrating display device 430 inaccordance with some embodiments. Display device 430 corresponds todisplay device 400 described herein with respect to FIG. 4A except that,in FIG. 4C, beam steerer 432 is integrated with combiner 410. Beamsteerer 432 is mechanically coupled with combiner 410 and configured torotate combiner 410 for directing light 414-1. In some embodiments,additionally or alternatively, beam steerer 432 is configured totranslate combiner 410 for directing light 414-1 (e.g., FIGS. 10A-10B).Light 414-1 projected by light projector 402 is received by combiner410, which, in turn, reflects light 414-1 toward pupil 406.

FIG. 4D is a schematic diagram illustrating display device 440 inaccordance with some embodiments. Display device 440 corresponds todisplay device 400 described herein with respect to FIG. 4A except that,in FIG. 4D, light projector 402 is integrated with eye tracker 408.Light 414-1, projected by light projector 402 is directed, by beamsteerer 404, toward combiner 410, which, in turn, reflects light 414-1toward pupil 406. Eye tracker 408 transmits ray 444-1 (e.g., an IR rayor a near-infrared ray) via the same optical path as light 414-1 towardpupil 406. Ray 444-1 is then reflected from pupil 406, via the sameoptical path, back to eye tracker 408, which detects the reflected ray444-1.

Although FIGS. 4A-4D illustrate display devices in accordance withvarious embodiments, one or more features described herein with respectto any one of FIGS. 4A-4D may be included in any of the display devicesdescribed herein with respect to any other drawings of FIGS. 4A-4D. Forexample, in some embodiments, the display devices described herein withrespect to FIGS. 4A-4C include light projector 402 integrated with eyetracker 408. In some embodiments, the display devices described hereinwith respect to FIGS. 4A and 4C-4D include light projector 402integrated with beam steerer 404. In some embodiments, the displaydevices described herein with respect to FIGS. 4A-4B and 4D includecombiner 410 integrated with beam steerer 432 in addition to, or insteadof, beam steerer 404. For brevity, such details are repeated herein.

FIGS. 5A-5B are schematic diagrams illustrating example operations ofdisplay device 400 in accordance with some embodiments.

In FIG. 5A, pupil 406 is located at first pupil position 406-1 directlyfacing combiner 410. The position of pupil 406 is determined by eyetracker 408, as illustrated with arrow 418-1. In some embodiments, theposition is determined based on pupil images, glint detection, detectionof retinal reflex (also called retinal reflection), or measurement of aprofile of a surface of pupil 406 (e.g., using a depth sensor). Inaccordance with a determination that pupil 406 is located at first pupilposition 406-1, light 414-1 projected by light projector 402 isdirected, by beam steerer 404, toward location 411-1 of combiner 410 sothat reflected light 414-1 is directed toward the pupil 406 at firstpupil position 406-1. In addition, combiner 410 transmits light 416 fromthe outside display device 400, and at least a portion of light 416 istransmitted toward pupil 406 at first pupil position 406-1.

In FIG. 5B, pupil 406 has moved to second pupil position 406-2 (e.g.,due to rotation of the eye). The position of pupil 406 is determined byeye tracker 408, as illustrated with arrow 418-2. In accordance with adetermination that pupil 406 is located at second pupil position 406-2,light 414-2 projected by light projector 402 is directed, by beamsteerer 404, toward location 411-2 of combiner 410 that is distinct fromlocation 411-1 of combiner 410. This causes combiner 410 to reflectlight 414-2 toward pupil 406 at second pupil position 406-2.

Although FIGS. 5A-5B illustrate steering the light projected by lightprojector 402 in one direction, a person having ordinary skill in theart would understand that the light projected by light projector 402 canbe steered in the opposite direction (e.g., from location 411-2 tolocation 411-1). For brevity, such details are not repeated herein.

Although FIGS. 5A-5B illustrate example operations of display device400, a person having ordinary skill in the art would understandanalogous operations applicable to any other display devices describedherein. For brevity, such details are not repeated herein.

Certain aspects of combiners are described herein with respect to FIGS.6A-6C, 7A-7B, 8, 9A-9E, and 10A-10B. Certain aspects of tracking aposition of pupil 406 are discussed herein with respect to FIGS.11A-11F. For brevity, such details are not repeated herein.

FIGS. 5C-5E are schematic diagrams illustrating display device 500 inaccordance with some embodiments.

Display device 500 in FIG. 5C corresponds to display device 400described herein with respect to FIG. 4A, except that light projector402, corresponding to light projector 402 of FIG. 4A positioned at afirst position, is mechanically coupled with actuator 502. In someembodiments, display device 500 does not include beam steerer 404, asshown in FIG. 5C. In some embodiments, display device 500 includes beamsteerer 404 in addition to actuator 502.

Actuator 502 is configured to change the position of light projector 402in one or more dimensions (e.g., three dimensions) by moving (e.g.,linearly or piecewise linearly) light projector 402, thereby changingthe optical path of light 514-1 projected by light projector 402. Insome embodiments, actuator 502 includes a voice coil motor, and movinglight projector 402 includes activating the voice coil motor. In someembodiments, actuator 502 is coupled with one or more translationstages, and moving light projector 402 includes activating the actuator502 to move the one or more translation stages.

In FIG. 5C, pupil 406 is located at first pupil position 406-1. Theposition of pupil 406 is determined by eye tracker 408. Also in FIG. 5C,light projector 402 is located at first position 402-1, and projectslight 514-1 to location 511-1 on combiner 410, where light 514-1 isdirected toward pupil 406 at first pupil position 406-1.

In some embodiments, light projector 402 is optically coupled with oneor more lenses 412 positioned on the optical path of light 514-1. Insome embodiments, one or more lenses 412 are mechanically coupled withactuator 502. In some embodiments, actuator 502 is configured to movelight projector 402 and one or more lenses 412 concurrently. In someembodiments, actuator 502 is configured to move light projector 402independent of one or more lenses 412 (e.g., one or more lenses 412 arenot mechanically coupled with actuator 502 so that actuator 502 moveslight projector 402 without moving one or more lenses 412).

In FIG. 5D, pupil 406 has moved to second pupil position 406-2 (e.g.,due to rotation of the eye). In response to a determination that pupil406 has moved to second pupil position 406-2, actuator 502 moves lightprojector 402 to second position 402-2, thereby directing light 514-2toward location 511-2 on combiner 410, where light 514-2 is directedtoward pupil 406 at second pupil position. In some embodiments, movementof light projector 402 does not change the direction of light projectedby light projector 402 (e.g., light 514-1 is parallel to light 514-2).In some embodiments, a direction of a movement of light projector 402from first position 402-1 to second position 402-2 is perpendicular to adirection of propagation of light 514-2. In some embodiments, actuator502 also moves one or more lenses 412 so that one or more lenses 412remains in the optical path of the light projected by light projector402.

In some embodiments, light projector 402 and/or one or more lenses 412are moved in a direction parallel to a direction of propagation of lightprojected by light projector 402 in order to adjust an image planecorresponding to the light projected by light projector 402. Thischanges a distance to a projected image, perceived by a wearer. In FIG.5E, pupil 406 remains in second pupil position 406-2. The image planecorresponding to the light projected by light projector 402 is adjustedby moving light projector 402 in a direction parallel to an optical axisof light projector 402 to third position 402-3. While light projector402 remains at third position 402-3, the image plane moves along anoptical axis in a direction toward light projector 402. In someembodiments, actuator 502 moves light projector 402 to a positionfurther away from pupil 406, which, in turn, moves the image plane alongthe optical axis in a direction away from light projector 402. By movinglight projector 402 back and forth in directions parallel to thedirection of propagation of light projected by light projector 402, theprojected images or objects appear to be closer or further away from awearer of display device 500.

Moving an image plane to change a perceived distance to a projectedimage is described further with respect to FIGS. 12A-12D.

FIG. 5F is a schematic diagram illustrating scanning reflector assembly515 in accordance with some embodiments. Scanning reflector assembly 515includes adjustable mirror 516 mechanically attached to actuator 502.Scanning reflector assembly 515 is optically coupled with lightprojector 402 (e.g., scanning reflector assembly 515 is configured toreceive light projected from light projector 402). Optionally, scanningreflector assembly 515 also includes mirror 518. In some embodiments,mirror 518 is either a stationary mirror or a movable mirror. In someembodiments, mirror 518 is also mechanically coupled with actuator 502(or an actuator that is separate from actuator 502). Scanning reflectorassembly 515 is configured to direct light from light projector 402toward a pupil of an eye of a wearer of a head-mounted display device.As the position of the pupil changes, the direction of the lightprojected by light projector 402 is changed by moving (e.g., by tilting)adjustable mirror 516 and/or mirror 518.

Although FIG. 5F illustrates scanning reflector assembly 515 withadjustable mirror 516 configured to rotate about one axis, in someembodiments, scanning reflector assembly 515 includes an adjustablemirror configured to rotate about two axes that are not parallel to eachother (e.g., the adjustable mirror is configured to rotate about thex-axis and also rotate about the y-axis at the same time and theadjustable mirror is also configured to rotate about the x-axis and alsorotate about the y-axis at separate times).

FIGS. 5G-5H are schematic diagrams illustrating display device 520 inaccordance with some embodiments. Display device 520 is similar todisplay device 500 described herein with respect to FIG. 5A, except thatdisplay device 520 includes scanning reflector assembly 515 opticallycoupled with light projector 402. In some embodiments, display device520 includes one or more lenses 412, which are not shown in FIG. 5G.

In FIG. 5G, pupil 406 is located at first pupil position 406-1. Theposition of pupil 406 is determined with eye tracker 408. Adjustablemirror 516 is located at first mirror position 516-1. Adjustable mirrordirects light 524-1 toward pupil 406 in first pupil position 406-1.

In FIG. 5H, pupil 406 has moved to second pupil position 406-2 (e.g.,due to a rotation of the eye). In response to a determination that thepupil has moved to second pupil position 406-2, adjustable mirror 516 istilted, by actuator 502, to second mirror position 516-2 so that light524-2, projected by light projector 402, is directed toward pupil 406 atsecond pupil position 406-2. In FIG. 5H, adjustable mirror is tiltedabout the x-axis. In some embodiments, adjustable mirror is tilted withrespect to the x-axis, the y-axis, and/or the z-axis.

FIG. 5I is a schematic diagram illustrating changing of direction ofprojected light in accordance with some embodiments. The light projectedby light projector (e.g., light 524-1 in FIG. 5G) has a cross-sectioncharacterized by two dimensions.

In some embodiments, the cross-section is symmetric with respect to thetwo dimensions (e.g., the cross-section is a square or a circle), asillustrated by cross-section 526-A in Section A of FIG. 5I. In someembodiments, the projected light is steered based on a position of apupil (e.g., first pupil position 406-1 and second pupil position 406-2in FIGS. 5G-5H) in two dimensions, as illustrated with cross-sections526-B and 526-C.

In some embodiments, the cross-section has a longitudinal shape (e.g., afirst dimension characterizing the cross-section, such as a length ofthe cross-section, is at least three times a second dimensioncharacterizing the cross-section, such as a width of the cross-section),as illustrated with cross-section 528 shown in Section B of FIG. 5I. Insome embodiments, the light is steered in only one dimension, asillustrated with an arrow in Section B.

In some embodiments, the direction of the light is changed by tiltingadjustable mirror 516 of scanning reflector assembly 515 in one, two, orthree dimensions, as described herein with respect to FIGS. 5F-5H.

In some embodiments, a direction of light projected by a light projector(e.g., light projector 402 described herein with respect to FIG. 4A) ischanged by beam steerer 432 integrated with combiner 410 (e.g., FIG.4C).

FIGS. 6A-6C and 7A-7B illustrate embodiments of flat combiners with beamsteering features in accordance with some embodiments.

FIGS. 6A-6C are schematic diagrams illustrating tunable waveguide 600 inaccordance with some embodiments. Tunable waveguide 600 includes opticalwaveguide 602 (e.g., a waveguide composed of a substrate of glass, fusedsilica or polycarbonate) configured to receive light (e.g., light 606)projected by light projector 402. In some embodiments, tunable waveguide600 includes one or more gratings coupled with optical waveguide 602 tofacilitate entry of light 606 into optical waveguide 602. The one ormore gratings are omitted in FIGS. 6A-6C so as not to obscure otheraspects of tunable waveguide 600.

In FIGS. 6A-6C, optical waveguide 602 is coupled with, or attached to, aplurality of tunable optical elements, such as tunable optical elements604-1 and 604-2. In some embodiments, tunable optical elements 604-1 and604-2 have electrically tunable optical properties. In some embodiments,tunable optical elements 604-1 and 604-2 includeindividually-addressable liquid crystal elements with electricallytunable indices of refraction. In some embodiments, individualadjustment of the index of refraction of tunable optical element 604-1causes light transmitted through optical waveguide 602 to emit fromoptical waveguide 602 at a location corresponding to tunable opticalelement 604-1.

In FIG. 6A, tunable optical elements 604-1 and 604-2 are under a firstoperating condition (e.g., both tunable optical elements 604-1 and 604-2are in a non-activated condition, such as no electric field is appliedto tunable optical elements 604-1 and 604-2). For example, in FIG. 6Aboth tunable optical elements 604-1 and 604-2 have a first index ofrefraction.

In FIG. 6A, the characteristics of light 606 are such that light 606propagates along optical waveguide 602 based on total internalreflection (TIR) (while tunable optical elements 604-1 and 604-2 remainin the first operating condition).

In FIG. 6A, light 606 continues to propagate through optical waveguide602, and light 606 is not emitted toward pupil 406 at first pupilposition 406-1.

In FIG. 6B, tunable optical elements 604-1 and 604-2 are under a secondoperating condition so that tunable optical element 604-2 iselectrically tuned to have a second index of refraction, distinct fromthe first index of refraction while tunable optical element 604-1continues to have the first index of refraction. After light 606propagates to a region of optical waveguide 602 adjacent to tunableoptical element 604-1, light 606 continues to propagate along opticalwaveguide 602. However, after light 606 propagates to a region ofoptical waveguide 602 adjacent to tunable optical element 604-2, light606 is emitted (e.g., escapes) from optical waveguide 602 toward pupil406 at first pupil position 406-1.

In FIG. 6C, pupil 406 has moved to second pupil position 406-2. In FIG.6C, tunable optical elements 604-1 and 604-2 are under a third operatingcondition so that tunable optical element 604-1 is electrically tuned tohave the second index of refraction (and tunable optical element 604-2may have the first index of refraction as shown in FIG. 6C or the secondindex of refraction). After light 606 propagates to a region of opticalwaveguide 602 adjacent to tunable optical element 604-1, light 606 isemitted (e.g., escapes) from optical waveguide 602 toward pupil 406 atsecond pupil position 406-2.

FIG. 7A is a schematic diagram illustrating holographic combiner 700 inaccordance with some embodiments. Holographic combiner 700 includesholographic optical element (HOE) 702. Holographic optical element 702includes one or more holographic films (e.g., a holographic film of aphotopolymer or an analog holographic film). In some embodiments, theone or more holographic films are located on substrate 708. Holographicoptical element 702 includes a plurality of portions (e.g., portions706-1, 706-2, and 706-3) configured to direct light in distinctdirections. In some embodiments, the portions of holographic opticalelement 702 are configured to direct light in distinct directions (e.g.,portion 706-1 of holographic optical element 702 is configured to directlight from light projector 402 into a first direction, portion 706-2 ofholographic optical element 702 is configured to direct light from lightprojector 402 into a second direction that is distinct from the firstdirection, and portion 706-3 of holographic optical element 702 isconfigured to direct light from light projector 402 into a thirddirection that is distinct from the first direction and the seconddirection). In some embodiments, holographic optical element 702 acts asa non-spherical reflective surface (e.g., a parabolic reflective surfaceor an ellipsoidal reflective surface) where rays impinging on thedifferent portions of holographic optical element 702 are directedtoward a pupil (e.g., pupil 406). In FIG. 7A, ray 704-1 projected bylight projector 402 impinges on portion 706-1, and portion 706-1 ofholographic optical element 702 is configured so that ray 704-1 isdirected toward pupil 406. Ray 704-2 projected by light projector 402impinges on portion 706-2, and portion 706-2 of holographic element 702is configured so that ray 704-2 is directed toward pupil 406. Ray 704-3projected by light projector 402 impinges on portion 706-3, and portion706-3 of holographic optical element 702 is configured so that ray 704-3is directed toward pupil 406. In some embodiments, holographic opticalelement 702 is a dynamically adjustable holographic optical element.

Although FIG. 7A illustrates three portions 706-1, 706-2, and 706-3 ofholographic optical element 702, in some embodiments, holographicoptical element 702 is configured to have more than three distinctportions. In some embodiments, a holographic combiner includes aplurality of distinct and separate holographic optical elements (e.g.,portions of the holographic optical element need not be continuous, insome embodiments).

FIG. 7B is a schematic diagram illustrating holographic combiner 710(also called a holographic waveguide combiner) in accordance with someembodiments. Holographic combiner 710 includes optical waveguide 714adjacent to holographic optical element 702. Waveguide 714 is configuredto receive light projected by light projector 402. In some embodiments,optical waveguide 714 corresponds to optical waveguide 602 describedherein with respect to FIG. 6A. For example, in some embodiments,optical waveguide 714 is coupled with one or more gratings to facilitateentry of the light projected by light projector 402 into opticalwaveguide 714. The one or more gratings are omitted in FIG. 7B so as notto obscure other aspects of holographic combiner 710.

In some embodiments, holographic combiner 710 also includes one or moreprisms (e.g., prism 712). Prism 712 is a tunable prism. In someembodiments, prism 712 is a tunable liquid prism. In some embodiments,prism 712 is a tunable liquid crystal prism. Tunable prism 712 isconfigured to dynamically steer light projected by light projector 402(e.g., rays 704-1, 702-2, and 704-3) based on a position of pupil 406.

FIG. 8 is a schematic diagram illustrating aspheric combiner 800 inaccordance with some embodiments. Aspheric combiner 800 includes apartially reflective aspheric surface (e.g., a parabolic surface, anellipsoidal surface, etc.) configured to receive rays 802-1, 802-2, and802-3 projected by light projector 402 and reflect at least a portion ofeach ray toward pupil 406.

In some embodiments, the partially reflective aspheric surface is awavelength-selective reflective surface (e.g., the partially reflectiveaspheric surface reflects a visible light of a first wavelength andtransmits a visible light of a second wavelength that is distinct formthe first wavelength). In some embodiments, the partially reflectiveaspheric surface includes wavelength-selective coatings, which aredescribed herein with respect to FIG. 9A. In such embodiments, asphericcombiner 800 reflects light with distinct wavelengths (e.g., blue, greenand/or red color) projected by light projector 402 while transmittingvisible wavelengths that are distinct from the wavelengths of lightprojected by light projector 402.

In some embodiments, the partially reflective surface is a polarizationdependent surface. Polarization dependent surfaces are described hereinwith respect to FIG. 10A. In such embodiments, aspheric combiner 800reflects light with a particular polarization (e.g., linearly polarizedlight) projected by light projector 402 while transmitting light havinga distinct polarization (e.g., circularly polarized light). In someembodiments, aspheric combiner 800 is mechanically coupled with anactuator for changing the position and/or orientation of the asphericcombiner in order to change the direction of rays 802-1, 802-2, and802-3 based on a position of pupil 406. For example, the position ofaspheric combiner 800 is moved in the z-direction and/or the orientationof aspheric combiner 800 changed by tilting aspheric combiner 800 withrespect to the x-axis.

FIG. 9A is a schematic diagram illustrating Fresnel combiner 900 inaccordance with some embodiments. In some embodiments, Fresnel combiner900 is an example of combiner 410 described herein with respect to FIG.4A. Fresnel combiner 900 is configured to combine light (e.g., light416) transmitted through combiner 900 from the outside of a head-mounteddisplay device with light (e.g., rays 910-1, 910-2 and 910-3) projectedby light projector 402. In some embodiments, light projector 402projects light with a distinct set of characteristics (e.g., light witha distinct polarization and wavelengths). In some embodiments, lightprojector 402 is configured to output two or more rays with distinctcharacteristics. For example, light projector 402 projects two or morelight rays with distinct colors (e.g., red, green, blue, infrared, etc.)and/or with distinct polarizations (e.g., a right hand polarization, aleft hand polarization, a horizontal polarization, a verticalpolarization, etc.). In some embodiments, light projector 402 includesor is optically coupled with one or more lenses 412 and/or beam steerer404 described herein with respect to FIG. 4A.

Fresnel combiner 900 includes substrate 902 with surface 902-1 facingpupil 406 and surface 902-2 opposite to surface 902-1. In someembodiments, surface 902-1 is a smooth surface and surface 902-2includes a plurality of Fresnel structures defined by a plurality ofdraft facets (e.g., draft facets 912-1, 912-2, and 912-3) and aplurality of slope facets (e.g., slope facets 904-1, 904-2, and 904-3).Draft facets 912-1, 912-2, and 912-3 are characterized by representativedraft angles (e.g., the draft facet is tilted by a respectiverepresentative draft angle from a reference axis). In some embodiments,draft facets 912-1, 912-2, and 912-3 are flat surfaces. In someembodiments, draft facets 912-1, 912-2, and 912-3 are curved surfaces,and the representative draft angle is an average draft angle for thedraft facet. In some embodiments, slope facets 904-1, 904-2, and 904-3are characterized by representative slope angles (e.g., the slope facetis tilted by a respective representative slope angle from the referenceaxis). In some embodiments, slope facets 904-1, 904-2, and 904-3 areflat surfaces. In some embodiments, slope facets 904-1, 904-2, and 904-3are curved surfaces, and the representative slope angle is an averageslope angle for the slope facet. In some embodiments, the shape of slopefacets 904-1, 904-2, and 904-3 is curved so that slope facets 904-1,904-2, and 904-3 correspond to segments of an aspheric surface (e.g., anellipsoidal surface or a parabolic surface).

Substrate 902 is made of an optically transparent material (e.g., glassor plastic). In some embodiments, at least a portion of surface 902-2includes a reflective coating. In some embodiments, at least a portionof surface 902-2 includes a semi-transparent optical coating thatreflects light (e.g., rays 910-1, 910-2, and 910-3) projected by lightprojector 402 while transmitting light (e.g., ambient light) fromopposite side of combiner 900 (e.g., light 416). In some embodiments,slope facets 904-1, 904-2, and 904-3 include a reflective opticalcoating while draft facets 912-1, 912-2, and 912-3 do not include areflective coating. In some embodiments, one or more slope facets do nothave a reflective coating (e.g., light impinging on such slope facetsare reflected by total internal reflection).

In some embodiments, the reflective coating is a wavelength-selectiveoptical coating. In some embodiments, slope facet 904-1 has awavelength-selective optical coating that reflects light with a specificwavelength while transmitting light with other wavelengths. For example,slope facet 904-1 has a wavelength-selective optical coating thatreflects red light with wavelength ranging from 625 nm to 675 nm whiletransmitting light outside of this range. The properties of thewavelength-selective coating are configured in accordance with thecharacteristics of light projected by light projector 402 (e.g., rays910-1, 910-2, and 910-3). For example, in a configuration where lightprojector 402 includes a narrow-bandgap light emitter (e.g., a laser ora superluminescent diode (SLD)), the reflective coating of slope facet904-1 is configured to reflect light with a narrow wavelength range(e.g., a wavelength range of 10 nm). If light projector 402 includes anLED, the reflective coating of slope facet 904-1 is configured toreflect light with a wider wavelength range (e.g., a wavelength range of50 nm). In some embodiments, slope facets 904-1, 904-2, and 904-3include a same wavelength-selective optical coating. In someembodiments, slope facet 904-1 has a wavelength-selective opticalcoating that is distinct from optical coatings of slope facets 904-2 or904-3. For example, slope facet 904-1 has a wavelength-selective opticalcoating that reflects red light (e.g., light with wavelength 625-675 nm)and slope facet 904-2 has a wavelength-selective optical coating thatreflects green light (e.g., light with wavelength 495-545 nm).

In some embodiments, the optical coatings of slope facets 904-1, 904-2,and 904-3 are polarization-selective optical coatings. For example,slope facet 904-1 has a coating that reflects light with a horizontalpolarization while transmitting through light with any otherpolarization and slope facet 904-2 has a coating that reflects lightwith a vertical polarization while transmitting through light with anyother polarization.

In FIG. 9A, light 416 from the outside of a head-mounted display deviceis transmitted through Fresnel combiner 900 toward pupil 406. In someembodiments, light 416 is refracted at surfaces 902-2 and 902-1. In someembodiments, Fresnel combiner 900 is optically coupled with otheroptical elements (e.g., one or more prisms) that are configured to steerlight 416 and/or rays 910-1, 910-2, and 910-3 (e.g., to compensate forthe refraction of light 416 at surfaces 902-2 and/or 902-1).

FIG. 9B is a schematic diagram illustrating Fresnel combiner 920 inaccordance with some embodiments. Fresnel combiner 920 includessubstrate 902 coupled with substrate 922. Substrate 922 includessurfaces 922-1 and 922-2. In some embodiments, surface 922-1 correspondsto surface 902-1 of substrate 902 (e.g., surface 922-1 is a smooth andflat surface). Surface 922-2 is configured to be an inverse replicate ofsurface 902-1, thereby including a plurality of Fresnel structures.Substrate 922 has an index of refraction corresponding to the index ofrefraction of substrate 902. In some embodiments, substrate 922 is madeof the same optically transparent material as substrate 902. Substrate902 and 922 are positioned so that there is a narrow space betweensubstrates 902 and 922 defined by respective surfaces 902-2 and 922-2.

In some embodiments, slope facets 904-1, 904-2, and 904-3 of substrate902 include wavelength-selective or polarization-selective coatings, asdescribed herein with respect to FIG. 9A. In such embodiments, rays910-1, 910-2, and 910-3 respectively corresponding to thecharacteristics of the coatings of slope facets 904-1, 904-2, and 904-3are reflected on the slope facets of surface 902-2, and transmittedtoward pupil 406, similarly as illustrated in FIG. 9A. However, in caseof Fresnel combiner 920 with two mating substrates 902 and 922, light416 is transmitted through Fresnel combiner 920 with only a slight shiftas refraction of light 416 on surface 922-2 is compensated by refractionof light 416 on surface 902-2.

FIG. 9C is a schematic diagram illustrating Fresnel combiner 930 inaccordance with some embodiments. Fresnel combiner 930 corresponds toFresnel combiner 920 described herein with respect to FIG. 9B exceptthat in Fresnel combiner 930 the space defined by surface 902-2 andsurface 922-2 is filled with material 932. Material 932 is composed ofan optically transparent material. In some embodiments, material 932 hasan index of refraction similar to an index of refraction of a materialused for substrates 902 and 922. In some embodiments, material 932 hasan index of refraction similar to an index of refraction of air.

FIG. 9D is a schematic diagram illustrating Fresnel combiner 940 inaccordance with some embodiments. Fresnel combiner 940 corresponds toFresnel combiner 920 described herein with respect to FIG. 9B or Fresnelcombiner 930 described herein with respect to FIG. 9C except thatFresnel combiner 940 additionally includes one or more optical elementsoptically coupled with surface 922-1 and/or surface 922-2. Such opticalelements further steer light 416 and/or rays 910-1, 910-2, and 910-3,and can be used to change the direction of the rays in accordance withrotation of the eye.

In FIG. 9D, Fresnel combiner 940 includes elements 942-1 and 942-2 thatare prisms. Element 942-1 is optically coupled with surface 922-1. Insome embodiments, element 942-1 is located adjacent to surface 922-1. Insome embodiments, element 942-1 is separated from optical surface 922-1.Optical element 942-2 is optically coupled with surface 902-1. In someembodiments, element 942-1 is located adjacent to surface 902-1. In someembodiments, element 942-1 is separated from optical surface 902-1. Insome embodiments, elements 942-1 and 942-2 are prisms made of anoptically transparent material (e.g., glass or plastic) (e.g.,transparent to visible light). In some embodiments, elements 942-1 and942-2 are liquid prims or liquid crystal prisms. In some embodiments,elements 942-1 and 942-2 are adaptive optical elements, such as adaptiveliquid prims or adaptive liquid crystal prisms. In some embodiments,elements 942-1 and 942-2 include one or more lenses, one or morediffraction gratings, one or more array of prims or any combinationthereof. In some embodiments, optical elements 942-1 and 942-2 furtherinclude coatings, such as wavelength-selective and/orpolarization-selective coatings.

FIG. 9E is a schematic diagram illustrating Fresnel combiner 950 inaccordance with some embodiments. Fresnel combiner 950 includessubstrate 902 and substrate 952. Substrate 952 is similar to substrate902 except substrate 952 includes slope facets 954-1, 954-2, and 954-3with optical coatings distinct from the optical coatings of slope facets904-1, 904-2, and 904-3. For example, slope facets 952-1, 954-2, and954-3 reflect light with distinct characteristics (e.g., color) fromlight reflected by slope facets 904-1, 904-2, and 904-3. In someembodiments, slope facets 904-1, 904-2, and 904-3 include awavelength-selective coating reflecting a first wavelength (e.g., ray910-1) while slope facets 954-1, 954-2, and 954-3 include awavelength-selective coating reflecting a second wavelength (e.g., ray910-4). For example, slope facets 904-1, 904-2, and 904-3 reflect greenlight while transmitting all other wavelengths and slope facets 954-1,954-2, and 954-3 reflect red light (and in some cases, transmit allother wavelength). In some embodiments, slope facets 904-1, 904-2, and904-3 include a polarization-selective coating reflecting light with afirst polarization while slope facets 954-1, 954-2, and 954-3 include apolarization coating reflecting light with a second polarization (e.g.,a polarization-selective coating for reflecting light with the secondpolarization). For example, slope facets 904-1, 904-2, and 904-3 reflectlight with a horizontal polarization while transmitting light having adifferent polarization and slope facets 954-1, 954-2, and 954-3 reflectlight with a vertical polarization while transmitting light having anyother polarization.

In FIG. 9E, substrate 952 is optically coupled with substrate 902 sothat they have the same optical axis. In some embodiments, substrate 952is adjacent to substrate 902. In some embodiments, substrates 952 and902 are configured differently (e.g., substrates 952 have Fresnelstructures that are different from Fresnel structures of substrate 902so that they have different focal lengths). In some embodiment,substrates 952 and 902 form a single contiguous substrate so thatsubstrate 952 and 902 are distinct portions of a single contiguousFresnel combiner. In some embodiments, Fresnel combiner 950 includesthree or more substrates optically coupled with each other for combininglight with distinct characteristics with ambient light. In someembodiments, substrate 952 is mechanically coupled with substrate 902.

FIG. 10A is a schematic diagram illustrating pancake combiner 1000 inaccordance with some embodiments. In some embodiments, pancake combiner1000 is an example of combiner 410 described herein with respect to FIG.4A. Pancake combiner 1000 is configured to combine light (e.g., light416) transmitted through combiner 1000 from the outside of ahead-mounted display device with light (e.g., ray 1008-1) projected bylight projector 402. Pancake combiner 100 is also configured to directray 1008-1 projected by light projector 402 toward an eye of a user(e.g., pupil 406). In FIG. 10A, pupil 406 is located at first pupilposition 406-1.

In some embodiments, light projector 402 is an example of lightprojector 402 described herein with respect to FIG. 4A. In someembodiments, light projector 402 projects polarized light. In someembodiments, light projector 402 includes one or more light sources thatemit linearly polarized light (e.g., a laser, or an LED). In someembodiments, light projector 402 includes one or more light sourcesemitting light with mixed polarization, and the one or more lightsources are optically coupled with a polarizer. In FIG. 10A, ray 1008-1projected by light projector 402 has a linear polarization. In someother embodiments, ray 1008-1 has a circular polarization. In someembodiments, light projector 402 is configured to emit light withdistinct wavelengths. In some embodiments, light projector 402 projectslight rays with blue, green and/or red color.

In FIG. 10A, light projector 402 is positioned away from an optical axisof pancake combiner 1000 and away from a path of light (e.g., light 416)transmitted through pancake combiner from the outside of a head-mounteddisplay. For example, light projector 402 is positioned on a temple of ahead-mounted display device. In some embodiments, light projector 402includes or is optically coupled with one or more lenses 412 and/or beamsteerer 404 described herein with respect to FIG. 4A. One or more lenses412 and beam steerer 404 are not shown in FIG. 10A so as not to obscureother aspects of pancake combiner 1000.

Pancake combiner 1000 includes partial reflector 1002, polarizer 1004and partial reflector 1006. The configuration of these opticalcomponents illustrated in FIG. 10A provides a folded optical path forray 1008-1, thereby allowing a smaller (e.g., thinner) combiner.

Partial reflector 1002 transmits at least a portion of ray 1008-1projected by light projector 402. In some embodiments, partial reflector1002 is a polarization dependent reflector that transmits light with aspecific polarization (e.g., a particular linear polarization). In someembodiments, partial reflector 1002 reflects light with any otherpolarization. In FIG. 10A, partial reflector 1002 transmits ray 1008-1with a first linear polarization, while reflecting light with a secondlinear polarization (e.g., perpendicular to the first linearpolarization). After ray 1008-1 is transmitted through partial reflector1002, ray 1008-1 is received by polarizer 1004, which converts thepolarization of ray 1008-1 from the first linear polarization to aright-handed circular polarization. In some embodiments, polarizer 1004is a quarter-wave plate.

Ray 1008-1 is subsequently reflected by partial reflector 1006. In someembodiments, partial reflector 1006 is a wavelength-selective mirrorreflecting light with one or more specific wavelengths whiletransmitting light with some other wavelengths. Wavelength-selectiveoptical coatings are described herein with respect to FIG. 9A. In FIG.10A, partial reflector 1006 is a wavelength-selective optical mirrorconfigured to reflect light with green color while transmitting throughlight with other colors, ray 1008-1 includes green light. In someembodiments, ray 1008-1 includes blue, green, and red light, and partialreflector 1006 is a wavelength-selective optical mirror reflecting blue,green, and red light while transmitting light with other colors. Whenpartial reflector 1006 reflects ray 1008-1, the circular polarizationshifts from a right-handed circular polarization to a left-handedcircular polarization. Ray 1008-1 reflected by partial reflector 1006 isreceived, for a second time, by polarizer 1004, which converts ray1008-1 to light with the second linear polarization.

Ray 1008-1 with the second linear polarization is reflected bypolarization dependent partial reflector 1002. Ray 1008-1 reflected bypartial reflector 1002 is received, for the third time, by polarizer1004, which converts ray 1008-1 to a light with a left-handed circularpolarization.

Ray 1008-1 is reflected, for the second time, by partial reflector 1006,and ray 1008-1 with a right-handed circular polarization is received,for the fourth time, by polarizer 1004. Polarizer 1004 converts ray1008-1 with the circular polarization to a light with the first linearpolarization. Ray 1008-1 with the first linear polarization istransmitted through partial reflector 1002 toward pupil 406.

In some embodiments, partial reflector 1002 is, in addition to beingpolarization dependent, also wavelength-selective, so that partialreflector 1002 only reflects light with wavelengths included in ray1008-1 (e.g., blue, green, and/or red color) while transmitting lightwith other wavelengths. Such wavelength-selective feature allows agreater amount of light from the outside of the display device (e.g.,light 416) to be transmitted through toward pupil 406. In someembodiments, partial reflector 1002 is polarization-independent and/orwavelength-independent. In some embodiments, partial reflector 1002 isconfigured to transmit a portion of impinging light and reflect aportion of the impinging light. For example, partial reflector 1002 is a50/50 mirror transmitting 50% and reflecting 50% of incident light(although a partial reflector having a different reflectance can beused, such as a 60/40 mirror, a 70/30 mirror, etc.). In someembodiments, partial reflector 1006 is polarization-independent and/orwavelength-independent. In some embodiments, partial reflector 1006 isconfigured to transmit a portion of impinging light and reflect aportion of the impinging light. For example, partial reflector 1006 is a50/50 mirror transmitting 50% and reflecting 50% of incident light(although a partial reflector having a different reflectance can beused, such as a 60/40 mirror, a 70/30 mirror, etc.). In someembodiments, partial reflector 1006 is a polarization dependent mirror.

In FIG. 10A, partial reflector 1006 is a parabolic partial mirrorconfigured to direct light projected by light projector 402 at a rangeof angles toward pupil 406. By adjusting the position and/or orientationof partial reflector 1006, the direction of ray 1008-1 emerging frompancake combiner 1000 is changed. In FIG. 10A, pupil 406 is located atfirst pupil position 406-1 and ray 1008-1 is directed toward pupil 406at first pupil position 406-1.

In FIG. 10B, pupil 406 has moved to second pupil position 406-2. Basedon the position of pupil 406, a head-mounted display shifts pancakecombiner 1000 (e.g., in the z-direction) so that ray 1008-2 is directedtoward pupil 406 at second pupil position 406-2. The direction of themovement and the distance of the movement are determined in accordancewith a position of pupil 406. In some embodiments, the orientation ofpancake combiner 1000 is adjusted to change the direction of ray 1008-2emerging from pancake combiner 1000. For example, the orientation ofpancake combiner 1000 is adjusted by tilting pancake combiner 1000 withrespect to a plane defined by pupil 406. In some embodiments, partialreflector 1006 has a curved shape other than a parabolic shape.

FIGS. 11A-11I illustrate methods and devices for eye tracking used inhead-mounted display devices in accordance with some embodiments. Eyetrackers illustrated in FIGS. 11A-11I are examples of eye tracker 408described herein with respect to FIG. 4A. The eye trackers areconfigured to determine a position of pupil 406 of eye 1100 (e.g., eye1100 is an eye of a wearer of a head-mounted display device) and/ortrack movement of pupil 406 as eye 1100 rotates to different gazedirections. In some embodiments, eye tracker 408 corresponds to, iscoupled with, or is included in eye tracking module 217 described hereinwith respect to FIG. 2.

FIG. 11A is a schematic diagram illustrating detecting a position of apupil by imaging in accordance with some embodiments. In FIG. 11A, eyetracker 408-1 includes a camera (e.g., a still camera or a video camera)that captures an image of an area surrounding pupil 406 (e.g., the areaof eye 1100 of a wearer of a head-mounted display device). In someembodiments, eye tracker 408-1 includes a telecentric camera. In FIG.11A, reference lines 1102 define an area (or a cone) imaged by eyetracker 408-1. The position of pupil 406 is determined from the capturedimage by determining positions of components of an eye (e.g., thesclera, the iris, and/or the pupil of the eye) by image processingalgorithms. In some embodiments, eye tracker 408-1 includes a lightsource for illumination of the imaged area. In some embodiments, thelight source emits infrared (IR) or near-infrared (NIR) and the cameracaptures an IR image or a NIR image of the eye.

FIG. 11B is a schematic diagram illustrating detecting a position of apupil by glint tracking in accordance with some embodiments. In FIG.11A, eye tracker 408-2 includes a camera (e.g., a still camera or avideo camera) or other photodetector (e.g., an array of photodiodes).Glint refers to a reflection of light from one or more surfaces of theeye. In FIG. 11B, eye tracker 408-2 projects light toward eye 1100 andat least a portion of the light, such as ray 1104-1, impinges on asclera of eye 1100 and is reflected as ray 1104-2. The reflected ray1104-2 is detected in the image captured by eye tracker 408-2.

In some embodiments, the position of pupil 406 is determined based on arepresentative intensity or intensities of detected rays. In someembodiments, the position of pupil 406 is determined based on anincident angle of detected ray 1104-2 (e.g., eye tracker 408-2 includesone or more optical elements to determine the incident angle of detectedray 1104-2). For example, the position of pupil 406 is determined bycomparing an incident angle of ray 1104-2 to an estimated surfaceprofile of surface of eye 1100. The surface profile of an eye does notcorrespond to a perfect sphere but instead has a distinct curvature inthe area that includes the cornea and the pupil. Therefore, a positionof the pupil can be determined by determining the surface profile of theeye.

In some embodiments, at least a portion of the light projected by eyetracker 408-1 impinges on other portions of eye 1100, such as ray 1106-1impinging on pupil 406. Ray 1106-1 is reflected as ray 1106-2, which isdetected in an image captured by eye tracker 408-2. In some embodiments,the position of pupil 406 is determined based on ray 1104-2 and ray1106-2. In some embodiments, the position of pupil 406 is determinedbased on a difference (and/or a ratio) between an intensity of ray1104-2 and an intensity of ray 1106-2. For example, the intensity of ray1104-2 reflected on the sclera of eye is higher than the intensity ofray 1106-2 and therefore the location of pupil 406 can be determinedbased on the intensity difference. In some embodiments, the position ofpupil 406 is determined based on an angle between ray 1104-2 and ray1106-2. For example, a surface profile of eye 1100 is determined basedon an angle between ray 1104-2 reflected on the sclera and ray 1106-2reflected on pupil 406.

In some embodiments, determining the position of pupil 406 by glinttracking includes projecting, by eye tracker 408-2, a light with astructured pattern (e.g., ray 1104-1 in FIG. 11B includes a structuredpattern) toward eye 1100 and detecting, by eye tracker 408-2, an imageof the structured pattern (e.g., ray 1104-2) as the light with thestructured pattern is reflected on the surface of eye 1100. For example,as a light with a structured pattern is reflected by the non-flatsurface of eye 1100, the structured pattern is distorted. The non-flatsurface profile of the eye 1100 is then determined based on thedistorted structured pattern and the position of pupil 406 is determinedbased on the surface profile.

FIG. 11C is a schematic diagram illustrating four structured patterns oflight used for eye tracking in accordance with some embodiments. Pattern1108-1 includes a straight line, pattern 1108-2 includes a grid pattern,pattern 1108-3 includes a dot matrix, and pattern 1108-4 includes asinusoid pattern.

FIG. 11D is a schematic diagram illustrating detecting a position of apupil by infrared (IR) retinal reflex detection in accordance with someembodiments. IR retinal reflex, also known as retinal reflection or redreflex, refers to a reddish reflection of light from the eye's retina,which is a layer of tissue that lines the inside of the back of an eye.The reflection of reddish light (e.g., infrared light) caused by the IRretinal reflex is distinguishable from reflection of light by otherpositions of the eye. For example, the light reflected from the retinahas a distinguishable intensity, wavelength dependence, and/orpolarization dependence. The retina includes the fovea, which is an arearesponsible for the sharp central vision including the highest densityof photoreceptors. Some photoreceptors (e.g., Henle fibers) havebirefringent properties. Birefringence refers to a property of amaterial to have a refractive index that depends on the polarizationstate and propagation direction of incident light. The birefringentproperties of the fovea can further be used to distinguish the lightreflected from the retina from light reflected from other portions ofthe eye.

Eye tracker 408-3 in FIG. 11D includes a light source (e.g., an IR lightsource) and an IR detector. In some embodiments, the IR detector is anIR-sensitive photodiode. In FIG. 11D, ray 1114 (e.g., an IR light) isprojected by eye tracker 408-3 toward eye 1100. As ray 1114 entersthrough the pupil, ray 1114 is reflected by the retina of eye 1100.Reflected ray 1116 is detected by eye tracker 408-3. A ray reflected bythe retina is distinguishable from light reflected from other portionsof the eye, and the location of the pupil can be determined based on theposition (and optionally an intensity) of the reflected light.

In some embodiments, eye tracker 408-3 is optically coupled with one ormore polarizers 1115. In some embodiments, polarizer 1115-1 ispositioned to polarize ray 1114 projected by eye tracker 408-3. Apolarized light impinging on the birefringent photoreceptors of theretina produces a reflection with a characteristic pattern.Alternatively or additionally, in some embodiments, polarizer 1115-2 ispositioned to polarize reflected ray 1116 detected by eye tracker 408-3,and a similar characteristic pattern is detected by eye tracker 408-3due to the birefringent photoreceptors. In such embodiments, theposition of pupil 406 of eye 1100 is determined based on thecharacteristic pattern detected by eye tracker 408-3 in reflected ray1116. In some embodiments, when no characteristic pattern is detected,eye tracker 408-3 determines that ray 1116 has been reflected by otherportions of eye 1100 (e.g., the sclera or the cornea).

FIG. 11E is a schematic diagram illustrating detecting a position of apupil by depth measurement in accordance with some embodiments. In someembodiments, eye tracker 408-4 includes one or more depth sensors (e.g.,one or more depth sensors based on a Time-of-Fly, Sound Navigation andRanging (SONAR), Light Detection and Ranging (LIDAR), interferometry,and/or light triangulation technique, and/or other techniques known inthe art for depth sensing). A depth sensor is configured to measure oneor more distances between eye tracker 408-4 and the surface of eye 1100(e.g., distances 1116-1, 1116-2, and 1116-3 in FIG. 11E). In someembodiments, a surface profile is determined by measuring two or more ofdistances 1116-1, 1116-2, and 1116-3. In some embodiments, the positionof pupil 406 is determined from the surface profile.

As eye 1100 rotates, the distance between eye tracker 408-4 and eye 1100changes. For example, in FIG. 11E, a distance longer than a referencedistance, such as a distance for an eye that is in a neutral position(e.g., 0°), indicates that the eye is looking down, and a distanceshorter than the reference distance indicates that the eye is lookingup. In addition, a gaze angle of eye 1100 can be determined from thedistance. Furthermore, an angular movement of eye 1100 can be determinedfrom changes in the distance. For example, in accordance with adetermination that the distance has increased, eye tracker 408-4determines that eye 1100 is rotating downward, and in accordance with adetermination that the distance has decreased, eye tracker 408-4determines that eye 1100 is rotating up. In FIG. 11E, eye tracker 408-4is configured to measure a distance between eye tracker 408-4 and asurface of eye 1100 along an axis that is not parallel to an opticalaxis of eye 1100 in a neutral position. For example, an optical axis ofeye tracker 408-4 is not aligned with an optical axis of eye 1100 in aneutral position (e.g., eye tracker 408-4 is positioned to measure adistance to a surface of eye 1100 at angle that is at least 15° awayfrom an optical axis of eye 1100 in a neutral position; eye tracker408-4 is positioned to measure a distance to a surface of eye 1100 atangle that is at least 30° away from an optical axis of eye 1100 in aneutral position; eye tracker 408-4 is positioned to measure a distanceto a surface of eye 1100 at angle that is at least 45° away from anoptical axis of eye 1100 in a neutral position; or eye tracker 408-4 ispositioned to measure a distance to a surface of eye 1100 at angle thatis at least 60° away from an optical axis of eye 1100 in a neutralposition).

FIG. 11F is a schematic diagram illustrating detecting a position of apupil by scanning in accordance with some embodiments. In someembodiments, an area including eye 1100 is scanned with eye tracker408-4 including a depth sensor to produce a contour profile of thesurface of eye 1100. In some embodiments, the contour profile of thesurface of the eye includes a three dimensional image of eye 1100. Insome embodiments, the three dimensional image of eye 1100 is a lowresolution image including only a limited number of depth measurementpoints sufficient to create an image of eye 1100. By limiting the numberof depth measurement points reduced time required of performing themeasurements as well as reducing the time and power required for imageprocessing of such images. This consequently reduces the powerconsumption required for eye tracking.

In some embodiments, an area including eye 1100 is scanned with anyother eye tracker described herein. In some embodiments, an eye trackerperforms a local search (e.g., scanning only a subset, less than all, ofthe area and determining a position of the pupil or a boundary of theiris), which improves the speed of eye tracking.

FIGS. 11G-11I are schematic diagrams illustrating detecting a positionof a pupil by a time-of-flight detector in accordance with someembodiments. Eye tracker 408-5 includes light source 1121 configured toemitting pulsed light toward eye 1100 and detector 1123 configured todetect pulsed light reflected by eye 1100.

In some embodiments, light source 1121 includes a light emitting diode(LED), a superluminescent light emitting diode (SLED), an organic LED(OLED) or a laser. In some embodiments, light source 1121 is a singleLED. In some embodiments, light source 1121 emits infrared (IR) ornear-infrared (NIR) light and detector 1123 is an IR or a NIR detector.In some embodiments, light source 1121 is included in, or is coupledwith, a light projector, such as light projector 402 described hereinwith respect to FIG. 4A. In FIGS. 11G-11H, eye tracker 408-5 emitspulsed light 1120-1 along optical path 1120 toward eye 1100. Pulsedlight 1120-1 is reflected by a surface of eye 1100 (e.g., the retina,the sclera, or the cornea of eye 1100) and the reflected light isdetected by detector 1123 of eye tracker 408-5. In some cases, pulsedlight 1120-1 is reflected by an area surrounding eye 1100 (e.g., an eyelid).

In some embodiments, detector 1123 is an avalanche photodiode (e.g., asingle-photon avalanche photodiode, SPAD). The avalanche photodiode isconfigured to receive light reflected by a surface of eye 1100 and eyetracker 408-5 is configured to determine a distance between the eyetracker 408-5 and the surface of eye 1100 which has reflected the lightbased on timing of emission and detection of light. In FIG. 11G, eye1100-1 is directed to a first angular position and pulsed light 1120-1projected toward eye 1100 enters eye 1100 through pupil 406 and isreflected back by the retina of eye 1100. Reflected pulsed light 1122-1returns along optical path 1122 and is detected by eye tracker 408-5.

In FIG. 11H, eye 1100-2 is directed to a second angular position andpulsed light 1120-1 projected toward eye 1100 does not enter eye 1100through pupil 406, and is reflected by the sclera of eye 1100. Reflectedpulsed light 1122-2 is detected by detector 1123.

In FIG. 11I, pulsed light 1120-1 emitted by light source 1121 andreflected light 1122-1 and 1122-2 are illustrated on a timeline. Pulsedlight 1120-1 is emitted at time T0 toward eye 1100 and time of emissionis recorded by eye tracker 408-5. In FIG. 11G, pulsed light 1120-1enters the pupil of eye 1100 and is reflected by the retina of eye 1100as pulsed light 1122-1. Pulsed light 1122-1 is detected by eye tracker408-5 at time T1, where T1-T0 corresponds to a time-of-flight for pulsedlight 1120-1 reflected by the retina of eye 1100. Based on thetime-of-flight T1-T0, a distance between eye tracker 408-5 and thesurface from which pulsed light 1122-1 has been reflected (e.g., theretina) is determined. In FIG. 11H, pulsed light 1120-1 is reflected bythe sclera of eye 1100 as pulsed light 1122-2. Pulsed light 1122-2 isdetected by eye tracker 408-5 at time T2, where T2-T0 corresponds to atime-of-flight of pulsed light 1120-1 reflected by the sclera of eye1100. Based on the time-of-flight T2-T0, a distance between eye tracker408-5 and the surface from which pulsed light 1120-1 has been reflected(e.g., the sclera) is determined. Based on the distances determinedbased on time-of-flights T1-T0 and T2-T0, eye tracker 408-5 determineswhether the time of flight corresponds to a time-of-flight of a pulsedlight reflected from the retina of eye 1100 or a time-of-flight of apulsed light reflected from the sclera of eye 1100.

FIG. 11J is a schematic diagram illustrating detecting a position of apupil by a time-of-flight detector in accordance with some embodiments.Eye tracker 408-6 includes a light source (e.g., light source 1124illustrated in the inset of FIG. 11J) emitting coherent light (e.g., ray1128) toward eye 1100. Eye tracker 408-6 also includes an interferometer(e.g., interferometer 1138 illustrated in the inset of FIG. 11J). Insome embodiments, the light source includes a light emitting diode(LED), a superluminescent light emitting diode (SLED), an organic LED(OLED) or a laser. In some embodiments, the light source is a singleLED. In some embodiments, the light source is included in a lightprojector such as light projector 402 described herein with respect toFIG. 4A. Ray 1128 is reflected by a surface of eye 1100. In FIG. 11J,ray 1128 enters eye 1100 through pupil 406 and is reflected by theretina of eye 1100. Reflected ray 1130 is subsequently detected byinterferometer 1138 of eye tracker 408-6.

The operation of interferometer 1138 is illustrated in the inset of FIG.11J. Light source 1124 emits ray 1132-1 impinging on beam splitter 1136.Beam splitter 1136 splits ray 1132-1 into ray 1128 directed toward eye1100 and reference ray 1132-2. Ray 1128 is reflected from a surface ofeye 1100. As described herein with respect to FIG. 11G, ray 1128 can bereflected by one or more surfaces of eye 1100 (e.g., the retina, thesclera, or the cornea), or surfaces surrounding eye 1100 (e.g., an eyelid). Reflected ray 1130 passes through beam splitter 1136 and entersdetector 1126 (e.g., a photodiode). Reference ray 1132-2 is reflected bymirror 1134, and a portion of the reflected reference ray 1132-3 isreflected by beam splitter 1136 toward detector 1126. Mirror 1134 is anadjustable mirror configured to change the optical distance traveled bythe reference ray 1132-3. Detector 1126 detects an interference patternformed by ray 1130 and reference ray 1132-3 arising from a phase-shiftbetween ray 1130 and reference ray 1132-3 due to a difference in thedistances traveled by the respective rays. Based on the interferencepattern, eye tracker 408-6 determines a distance between eye tracker408-6 and a surface which reflected ray 1128 is determined. Based on thedistance between eye tracker 408-6 and the surface of which ray 1128,eye tracker 408-6 determines whether ray 1128 is reflected by the retinaor any other surfaces of the eye (e.g., the sclera or the cornea of eye1100).

In some embodiments, the eye trackers described herein with respect toFIGS. 11A-11J (i.e., eye trackers 408-1, 408-2, 408-3, 408-4, 408-5, and408-6) are configured to obtain a low-resolution image of eye 1100 foreye tracking. In some embodiments, the low-resolution image includes asingle-pixel image. In some embodiments, the low-resolution imageincludes an image consisting of a few pixels (e.g., 2, 3, 4, 5, 6, 7, 8,9, 10 pixels, etc.). In some embodiments, the eye trackers of FIGS.11A-11J include a single-pixel eye tracking sensor. In some embodiments,the eye trackers of FIGS. 11A-11J include a low-resolution image sensorsuch as a small array of eye tracking sensors (e.g., 2, 3, 4, 5, 6, 7,8, 9, 10 eye tracking sensors, etc.). A single-pixel sensor and alow-resolution image sensor have advantages over eye trackers includinga camera providing high-resolution images (e.g., 500×500 pixels or more,etc.). For example, a single-pixel eye tracking sensor has low powerconsumption and a low weight compared to a high-resolution camera. Also,processing data from a single-pixel sensor requires less computationalpower than processing data from a high-resolution image.

FIGS. 12A-12E are schematic diagrams illustrating head-mounted displayswith focus-supporting light projectors in accordance with someembodiments.

Vergence-accommodation conflict can impact the quality of userexperience with head-mounted displays. Focus-supporting lightprojectors, unlike fixed-focus light projectors, are capable of changinga perceived distance to a projected image, thereby reducing oreliminating vergence-accommodation conflict.

FIG. 12A illustrates display device 1210 in accordance with someembodiments. Display device 1210 is similar to display device 500 shownin FIG. 5D, except that, in FIG. 12A, actuator 502 is configured to moveone or more lenses 412 without moving light projector 402. In FIG. 12A,actuator 502 is configured to move one or more lenses 412 in a directionthat includes a component parallel to an optical axis of light projector402 (e.g., actuator 502 moves one or more lenses 412 along the opticalaxis of light projector 402). This allows moving an image planecorresponding to the projected images. Display device 1210 optionallyincludes beam steerer 404.

FIG. 12B illustrates display device 1220 in accordance with someembodiments. Display device 1220 is similar to display device 1210 shownin FIG. 12A, except that, in FIG. 12B, actuator 502 is configured tomove light projector 402 (or a light source within light projector 402)without moving one or more lenses 412. In FIG. 12B, actuator 502 isconfigured to move light projector 402 in a direction that includes acomponent parallel to an optical axis of light projector 402 (e.g.,actuator 502 moves light projector 402 along the optical axis of lightprojector 402).

In some embodiments, actuator 502 is configured to move one or morelenses 412 along with light projector 402, as shown in FIG. 5D.

FIG. 12C illustrates display device 1230 in accordance with someembodiments. Display device 1230 is similar to display device 1210 shownin FIG. 12A, except that display device 1230 includes one or morespatial light modulators 1202. In some embodiments, one or more spatiallight modulators 1202 include a liquid-crystal-on-silicon (LCOS) spatiallight modulator. In some embodiments, one or more spatial lightmodulators 1202 include a phase-modifying spatial light modulator. Thephase-modifying spatial light modulator functions like a tunable lenshaving a variable focal length.

FIG. 12D illustrates display device 1240 in accordance with someembodiments. Display device 1240 is similar to display device 1230 shownin FIG. 12C, except that display device 1240 includes tunable lens 1204,such as an electro-wetting lens, a liquid lens, and a liquid crystallens. In some embodiments, display device 1240 includes one or morelenses 412. In some embodiments, display device 1240 does not includeone or more lenses 412.

FIG. 12E illustrates display device 1250 in accordance with someembodiments. Display device 1250 is an example of a Maxwellian-viewhead-mounted display. Display device 1250 includes blocker 1206 definingan aperture (e.g., a pinhole having a diameter 0.1 mm, 0.2 mm, 0.3 mm,0.4 mm, 0.5 mm, 0.6 mm, 0.7 mm, 0.8 mm, 0.9 mm, 1 mm, etc.). In someembodiments, blocker 1206 is made of an opaque material (e.g., metal,plastic, etc.) configured to block light (and transmit light onlythrough the aperture). Light projector 402 is optically coupled with theaperture so that light from light projector 402 is transmitted throughthe aperture defined in blocker 1206. In some embodiments, one or morelenses 412 are configured to focus the light transmitted through theaperture onto (or adjacent to) pupil 406.

In some embodiments, display device 1250 includes one or more spatiallight modulators (e.g., an amplitude-modifying spatial light modulator)configured to modify the light transmitted through the aperture.

FIGS. 12F and 12G are schematic diagrams illustrating example operationsof a head-mounted display in accordance with some embodiments.

FIG. 12F illustrates that eyes are gazing at object 1270 (e.g., a realobject or a virtual object) located adjacent to the user. As shown inFIG. 12F, when the eyes are gazing an object that is located adjacent tothe user, the eyes converge (as compared to their neutral positions).Eye trackers 408-1 and 408-2 determine positions of pupils of the eyes.In some embodiments, eye trackers 408-1 and 408-2 (or one or moreprocessors in communication with eye trackers 408-1 and 408-2, such asprocessors 216) determine a vergence based on the positions of thepupils of the eyes (e.g., by determining gaze angles of the eyes andcalculating a difference between the gaze angles). In some embodiments,the one or more processors determine an image plane based on thevergence. For example, the one or more processors select an adjacentimage plane (e.g., an image plane having a first distance from the user)based on the convergence of the eyes, and select a far image plane(e.g., an image plane having a second distance, greater than the firstdistance, from the user) based on the divergence of the eyes.

Display devices 1260-1 and 1260-2 project light for rendering images. Insome embodiments, each of display devices 1260-1 and 1260-2 includes afocus-supporting light projector described herein (e.g., thefocus-supporting light projector of display device 1250). In FIG. 12F,display device 1260-1 projects light, and combiner 410-1 reflects theprojected light toward the left eye. In addition, display device 1260-2projects light, and combiner 410-2 reflects the projected light towardthe right eye. In some embodiments, the light projected by displaydevice 1260-1 is different from the light projected by display device1260-2. In some cases, this provides a stereoscopic perception to theuser.

In FIG. 12F, the light projected by display device 1260-1 is illustratedwith two representative rays 1262-1 and 1262-2, and the light projectedby display device 1260-2 is illustrated with two representative rays1264-1 and 1264-2. Rays 1262-1 and 1264-1 correspond to object 1270 thatis located adjacent to the user, and rays 1262-2 and 1264-2 correspondto object 1272 that is located away from the user.

While the user is gazing at object 1270 that is located adjacent to theuser, rays 1262-1 and 1264-1 are adjusted (e.g., using afocus-supporting light projector, such as display device 1250) to remainin focus when projected onto the retinas of the eyes and rays 1262-2 and1264-2 are adjusted (e.g., using the focus-supporting light projector)to remain out of focus when projected onto the retinas. This creates avisual perception that the object located adjacent to the user appearsin focus and the object located away from the user appears blurry.

FIG. 12G illustrates that the eyes are gazing at object 1272 (e.g., areal object or a virtual object) located away from the user (e.g.,object 1270 is perceived to be located at a first distance from theuser, and object 1272 is perceived to be located at a second distance,from the user, that is greater than the first distance). Eye trackers408-1 and 408-2 determine positions of pupils of the eyes. In someembodiments, eye trackers 408-1 and 408-2 determine a vergence based onthe positions of the pupils of the eyes.

Display devices 1260-1 and 1260-2 project light for rendering updatedimages. For example, rays 1262-1 and 1264-1 are adjusted to remain outof focus when projected onto the retinas and rays 1262-2 and 1264-2 areadjusted to remain in focus when projected onto the retinas. Thiscreates a visual perception that the object located adjacent to the userappears blurry and the object located away from the user appears infocus.

In light of these principles, we now turn to certain embodiments ofhead-mounted display devices.

In accordance with some embodiments, a head-mounted display device forproviding augmented reality contents to a wearer includes an eyetracker, a light projector, a beam steerer and a combiner (e.g., displaydevice 400 includes eye tracker 408, light projector 402, beam steerer404 and combiner 410 in FIG. 4A). The eye tracker is configured todetermine a position of a pupil of an eye of the wearer (e.g., aposition of pupil 406 in FIG. 4A), the light projector is configured toproject light (e.g., light 414-1) for rendering images based at least onthe augmented reality contents, the beam steerer is configured to changea direction of the light (e.g., light 414-1) from the light projectorbased on the position of the pupil, and the combiner is configured tocombine the light from the light projector and light from an outside ofthe head-mounted display device for providing an overlap of the renderedimage and a real image that corresponds to the light from the outside ofthe head-mounted display device (e.g., combiner 410 combines light 414-1projected by light projector 402 with light 416 from the outside ofdisplay device 400 (e.g., ambient light) in FIG. 4A).

In some embodiments, the eye tracker includes any eye tracker describedherein (e.g., FIGS. 11A-11J).

In some embodiments, the beam steerer includes any beam steerer or anyactuator described herein (e.g., FIGS. 4A-4D and 5A-5I).

In some embodiments, the light projector is a focus-supporting lightprojector (e.g., FIGS. 12A-12E).

In some embodiments, the combiner is any combiner described herein(e.g., FIGS. 6A-6C, 7A-7B, 8, 9A-9E, and 10A-10B).

In some embodiments, the light projector is configured to project thelight over an area that is within a 6 mm×6 mm square area on a plane ofthe pupil of the eye of the wearer (e.g., an area within a 6 mm×6 mmsquare area on reference plane 407-1 of pupil 406 in FIG. 4A).

In some embodiments, the light projector is configured to project thelight over an area with a characteristic dimension of at least 3 mm.This reduces or eliminates underfilling of a pupil.

In some embodiments, the combiner does not include a grating coupledwith a waveguide (e.g., combiner 410 in FIG. 4A does not include agrating coupled with a waveguide, which are used in a pupil replicatingcombiner). In some embodiments, the combiner is not configured to expandthe size of a beam using pupil replication.

In some embodiments, the light projector includes one or more of: aspatial light modulator (e.g., FIG. 12C) or a scanning mirror projector(e.g., FIG. 5G).

In some embodiments, the light projector is coupled with one or moreadjustable focus lenses (e.g., lens 1204 in FIG. 12D).

In some embodiments, the light projector is coupled with an aperture andone or more lenses (e.g., Maxwellian optics shown in FIG. 12E).

In some embodiments, the beam steerer includes one or more actuators(e.g., actuator 502 in FIG. 5C).

In some embodiments, the combiner includes one or more of: a Fresnelcombiner (e.g., FIGS. 9A-9E), a pancake combiner (e.g., FIGS. 10A-10B),an ellipsoidal mirror (e.g., FIG. 8), one or more tunable waveguides(e.g., FIGS. 6A-6C), or a holographic combiner (e.g., FIGS. 7A and 7B).

In some embodiments, the combiner is configured to reflect the lightfrom the light projector and transmit the light from the outside of thehead-mounted display device (e.g., in FIG. 4A, combiner 410 reflectslight 414-1 projected by light projector 402 and transmits light 416from the outside of display device 400).

In some embodiments, the eye tracker includes one or more of: a pupilimage tracker (e.g., FIG. 11A), a retinal reflux tracker (e.g., FIG.11D), a depth sensor (e.g., FIG. 11E), or a glint tracker (e.g., FIG.11B).

In some embodiments, the beam steerer is integrated with the lightprojector (e.g., in FIG. 4B, component 422 includes beam steerer 404integrated with light projector 402).

In some embodiments, the beam steerer is integrated with the lightcombiner (e.g., in FIG. 4C, beam steerer 432 is integrated with combiner410).

In some embodiments, the eye tracker is integrated with the lightprojector (e.g., in FIG. 4D, eye tracker 408 is integrated with lightprojector 402).

In accordance with some embodiments, a method providing augmentedreality contents to a wearer using a head-mounted display device (e.g.,display device 400 in FIG. 5A) that includes an eye tracker, a lightprojector, a beam steerer, and a combiner includes determining aposition of a pupil of an eye with the eye tracker (e.g., position ofpupil 406 is determined by eye tracker 408 in FIG. 4A). The method alsoincludes projecting, with the light projector, light for renderingimages based at least on the augmented reality contents and changing,with the beam steerer, a direction of the light from the light projectorbased on the position of the pupil (e.g., beam steerer 404 changes thedirection of beam 414-1 directed toward pupil 406 at a first position inFIG. 5A so that beam 414-2 is directed toward pupil 406 at a secondposition as shown in FIG. 5B). The method further includes combining,with the combiner, the light from the light projector and light from anoutside of the head-mounted display device for providing an overlap ofthe rendered image and a real image that corresponds to the light fromthe outside of the head-mounted display device (e.g., combiner 410combines rays 414-1 and 416 in FIG. 5A).

In some embodiments, the method includes projecting, with the lightprojector, the light over an area that is within a 6 mm×6 mm square areaon a plane of the pupil of the eye of the wearer.

In some embodiments, the method includes projecting, with the lightprojector, the light over an area with a characteristic dimension of atleast 3 mm.

In some embodiments, the combiner does not include a grating coupledwith a waveguide.

In some embodiments, the light projector includes one or more adjustablefocus lenses.

In some embodiments, the light projector includes the light projector iscoupled with an aperture and one or more lenses.

In accordance with some embodiments, a method for providing images to awearer using a head-mounted display device including a light projectorand a beam shifter (e.g., display device 500 includes light projector402 and actuator 502 in FIG. 5C) includes projecting, with the lightprojector, light for rendering images (e.g., light 514-1) based at leaston virtual reality contents and/or augmented reality contents andchanging, with a beam shifter, a path of the light projected from thelight projector based on a position of a pupil (e.g., pupil 406) of aneye of the wearer.

In some embodiments, the beam shifter is mechanically coupled with thelight projector and changing the path of the light from the lightprojector includes moving, with the beam shifter, the light projector ina direction that is non-parallel to an optical axis of the lightprojector. For example, actuator 502 is mechanically coupled with lightprojector 402 and moving the light projector 402 changed the directionof light 514-1 in FIG. 5C).

In some embodiments, the method includes moving the light projector in afirst direction that is non-parallel to the optical axis of the lightprojector at a first time, and moving the light projector in a seconddirection that is non-parallel to the first direction and the opticalaxis of the light projector at a second time. For example, lightprojector 402 in FIG. 5C is moved, non-parallel to the optical axis oflight projector 402, to a second position indicated with light projector402 in FIG. 5C).

In some embodiments, the head-mounted display device includes areflector configured to receive the light from the light projector anddirect the light toward the pupil of the eye of the wearer. The beamshifter is mechanically coupled with the reflector, and changing thepath of the light includes moving, with the beam shifter, the reflector.For example, scanning reflector 515 includes adjustable mirror 516mechanically coupled with actuator 502, and by moving scanning reflector515 with actuator 502 changes the direction of light projected by lightprojector 402 in FIG. 5F.

In some embodiments, the light projected from the light projector has across-section that is characterized by a first dimension and a seconddimension that is shorter than the first dimension (e.g., cross-section528 has a longitudinal shape in Section B of FIG. 5I). Moving thereflector includes tilting the reflector about a first axis withouttilting the reflector about a second axis that is non-parallel to thefirst axis (e.g., cross-section 528 in Section B of FIG. 5I is moved inone dimension by tilting adjustable mirror 516 in FIG. 5G in only onedirection).

In some embodiments, moving the reflector includes tilting the reflectorabout a first axis at a first time and tilting the reflector about asecond axis that is non-parallel to the first axis at a second time(e.g., cross-section 526-A in Section A of FIG. 5I is moved in twodimensions by tilting adjustable mirror 516 in FIG. 5G in twodirections).

In some embodiments, the method includes determining, with an eyetracker (e.g., eye tracker 408 in FIG. 5C) coupled with the head-mounteddisplay device, a position of the pupil of the eye of the wearer.

In accordance with some embodiments, a head-mounted display device forproviding images to a wearer includes a light projector configured toproject light for rendering images based at least on virtual realitycontents and/or augmented reality contents and a beam shifter configuredto change a path of the light projected from the light projector basedon a position of a pupil of an eye of the wearer (e.g., FIG. 5C).

In some embodiments, the beam shifter is mechanically coupled with thelight projector, and the beam shifter is configured to change the pathof the light from the light projector by moving the light projector in adirection that is non-parallel to an optical axis of the light projector(e.g., FIG. 5C).

In some embodiments, the beam shifter is configured to move the lightprojector in a first direction that is non-parallel to the optical axisof the light projector at a first time (e.g., FIGS. 5C and 5D). The beamshifter is also configured to move the light projector in a seconddirection that is non-parallel to the first direction and the opticalaxis of the light projector at a second time.

In some embodiments, the beam shifter is configured to move the lightprojector in a direction that is parallel to the optical axis of thelight projector (e.g., light projector 402 at a second position in FIG.5D is moved, parallel to the optical axis of light projector 402, to athird position 402-3 of light projector 402 shown in FIG. 5E).

In some embodiments, the head-mounted display device includes a lensoptically coupled with the light projector, and the device furtherincludes a beam shifter configured to move the lens in a direction thatincludes a component parallel to the optical axis of the light projector(e.g., one or more lenses 412 at a first position in FIG. 5C is moved,parallel to light projector 402, to a second position in FIG. 5D).

In some embodiments, the device includes a holographic combinerconfigured to receive the light projected from the light projector anddirect the light toward the pupil of the eye of the wearer. Theholographic combiner includes a holographic optical element located on atransparent substrate (e.g., holographic combiner 700 includesholographic optical element 702 on substrate 708 in FIG. 7A). Theholographic optical element includes a first portion (e.g., portion706-1) configured to direct a light (e.g., ray 704-1), from the lightprojector, impinging on the first portion of the holographic opticalelement in a first direction (e.g., toward pupil 406). The holographicoptical element also includes a second portion (e.g., portion 706-2)configured to direct a light (e.g., ray 704-2), from the lightprojector, impinging on the second portion of the holographic opticalelement in a second direction (e.g., toward pupil 406) that is distinctfrom the first direction. The holographic optical element furtherincludes third portion (e.g., portion 706-3) configured to direct alight (e.g., ray 704-3), from the light projector, impinging on thethird portion of the holographic optical element in a third direction(e.g., toward pupil 406) that is distinct from the first direction, thesecond direction, and the third direction.

In some embodiments, the head-mounted display device includes areflector (e.g., adjustable mirror 516-1 in FIG. 5G) configured toreceive the light from the light projector and direct the light towardthe pupil of the eye of the wearer. The beam shifter (e.g., actuator502) is mechanically coupled with the reflector, and the beam shifter isconfigured to change the path of the light by moving the reflector.

In some embodiments, the light projected from the light projector has across-section that is characterized by a first dimension and a seconddimension that is shorter than the first dimension (e.g., FIG. 5G andFIG. 5I). The beam shifter is configured to move the reflector bytilting the reflector about a first axis without tilting the reflectorabout a second axis that is non-parallel to the first axis.

In some embodiments, the beam shifter is configured to move thereflector by tilting the reflector about a first axis at a first timeand tilting the reflector about a second axis that is non-parallel tothe first axis at a second time (e.g., FIG. 5G and FIG. 5I).

In some embodiments, the head-mounted display device is coupled with aneye tracker configured to determine a position of the pupil of the eyeof the wearer (e.g., eye tracker 408 in FIG. 5G).

In some embodiments, the beam shifter includes a tunable waveguide(e.g., tunable waveguide 600 in FIGS. 6A-6C). The tunable waveguideincludes a waveguide (e.g., waveguide 602) configured to receive thelight projected from the light projector (e.g., light 606). Thewaveguide also includes a plurality of individually-addressable controlregions located adjacent to the waveguide along the waveguide (e.g.,tunable optical elements 604-1 and 604-2). A respective region of theplurality of individually-addressable control regions is configured tohave a first optical thickness under a first operating condition and asecond optical thickness that is distinct from the first opticalthickness under a second operation condition that is distinct from thefirst operating condition (e.g., tunable optical element 604-1 has afirst optical thickness in FIG. 6A and a second optical thickness inFIG. 6C). The tunable waveguide is configured to propagate the receivedlight through the waveguide at a location that corresponds to therespective region while the respective region has the first opticalthickness (e.g., in FIG. 6A, light 606 is propagates through waveguide602 after impinging on tunable optical element 604-1 with the firstoptical thickness) and emit at least a portion of the received lightfrom the waveguide at the location that corresponds to the respectiveregion while the respective region has the second optical thickness(e.g., in FIG. 6C, light 606 is emitted from waveguide 602 afterimpinging on tunable optical element 604-1 with the second opticalthickness).

In some embodiments, the beam shifter includes a holographic combiner(e.g., holographic combiner 710 includes waveguide 714 in FIG. 7B). Theholographic combiner includes a waveguide configured to receive thelight projected from the light projector. The holographic combiner alsoincludes one or more holographic optical elements (e.g., holographicoptical element 702) located adjacent to the waveguide and one or moretunable prisms (e.g., prism 712) located adjacent to the one or moreholographic optical elements.

In some embodiments, the one or more tunable prisms include one or moreof: a liquid prism or a liquid crystal prism (e.g., prism 712 in FIG.7B).

In accordance with some embodiments, a head-mounted display device(e.g., display device 100 in FIG. 1) for providing augmented realitycontents to a wearer includes a first light projector configured toproject light for rendering images based at least on the augmentedreality contents (e.g., light projector 402 in FIG. 4A), and a firstFresnel combiner (e.g., Fresnel combiner 900 in FIG. 6A) configured tocombine the light from the first light projector (e.g., rays 910-1,910-2, and 910-3 in FIG. 6A) and light from an outside of thehead-mounted display device (e.g., light 416 in FIG. 6A) for providingan overlap of the rendered image and a real image that corresponds tothe light from the outside of the head-mounted display device.

In some embodiments, the first Fresnel combiner includes a firstoptically transparent substrate having a first surface and a secondsurface that is opposite to the first surface (e.g., Fresnel combiner900 includes substrate 902 with surface 902-1 and 902-2 in FIG. 6A). Thefirst Fresnel combiner includes a plurality of Fresnel structures on thesecond surface (e.g., surface 902-2 includes Fresnel structures definedby slope facets 904-1, 904-2, and 904-3, and adjacent draft facets912-1, 912-2, and 912-3).

In some embodiments, the first Fresnel combiner has no Fresnel structureon the first surface (e.g., surface 902-1 is a smooth and flat surfacein FIG. 9A).

In some embodiments, the first Fresnel combiner is configured to reflectthe light from the first light projector and transmit the light from theoutside of the head-mounted display device (e.g., rays 910-1, 910-2, and910-3 emitted by light projector 402 are reflected at surface 902-2 andlight 416 from the outside of the head-mounted display 416 istransmitted through Fresnel combiner 900 in FIG. 9A).

In some embodiments, the first light projector is positioned away from apath of the light, from the outside of the head-mounted display device,transmitted through the first Fresnel combiner (e.g., light projector402 is positioned away from light 416 in FIG. 9A).

In some embodiments, the first light projector is located away from anoptical axis of the first Fresnel combiner (e.g., light projector 402 islocated away from an optical axis of Fresnel combiner 900 in FIG. 9A).

In some embodiments, the first Fresnel combiner includes one or morewavelength-selective optical coatings on at least a portion of thesecond surface (e.g., slope facets 904-1, 904-2, and 904-3 of Fresnelcombiner 900 include optical coatings in FIG. 9A).

In some embodiments, the one or more wavelength-selective opticalcoatings include at least one optical coating that has a first index ofrefraction for light of a first color and a second index of refraction,distinct from the first index of refraction, for light of a second colorthat is distinct from the first color so that the first Fresnel combinerreflects the light of the first color projected from the first lightprojector and forgoes reflecting the light of the second color projectedfrom the first light projector. For example, in FIG. 9A optical coatingof slope facet 904-1 has a selective index of refraction for firstwavelength range (e.g., red color) and thereby reflects light within thefirst wavelength range while transmitting light with wavelength outsidethe first wavelength range. Optical coating of slope facet 904-2 has aselective index of refraction for a second wavelength range (e.g., greencolor) and thereby reflects light within the second wavelength rangewhile transmitting light with wavelength outside the second wavelengthrange.

In some embodiments, the first Fresnel combiner includes a secondoptically transparent substrate having a third surface and a fourthsurface that is opposite to the third surface, and the first Fresnelcombiner includes a plurality of Fresnel structures on the third surface(e.g., substrate 922 includes surface 922-1 and surface 922-2, which hasa plurality of Fresnel structures in FIG. 9B).

In some embodiments, the first Fresnel combiner has no Fresnel structureon the fourth surface (e.g., surface 922-1 is smooth and flat surface inFIG. 9B).

In some embodiments, the plurality of Fresnel structures on the thirdsurface is configured to mate with the plurality of Fresnel structureson the second surface (e.g., surface 922-2 is configured to mate withsurface 902-2 in FIG. 9B).

In some embodiments, the first optically transparent substrate and thesecond optically transparent substrate are made of a first materialhaving a first index of refraction (e.g., substrates 902 and 922 in FIG.9B). The plurality of Fresnel structures on the second surface isseparated from the plurality of Fresnel structures on the third surface(e.g., surfaces 902-2 and 922-2 define a spacing between them). Aspacing between the plurality of Fresnel structures on the secondsurface and the plurality of Fresnel structures on the third surface isfilled with a second material having a second index of refraction thatis less than the first index of refraction (e.g., in some embodiments,the spacing defined by surfaces 902-2 and 922-2 is filled with anoptically transparent material with different index of refraction thanthe index of refraction of substrates 902 and 922 in FIG. 9C).

In some embodiments, the first Fresnel combiner is configured to reflectthe light from the first light projector by total internal reflectionwithin the first optically transparent substrate.

In some embodiments, the first Fresnel combiner is configured totransmit the light from the outside of the head-mounted display device(e.g., light 416 in FIG. 9B) through the first optically transparentsubstrate, the second material in the spacing between the plurality ofFresnel structures on the second surface and the plurality of Fresnelstructures on the third surface, and the second optically transparentsubstrate.

In some embodiments, the head-mounted display device further includesone or more prisms optically coupled with the first surface of the firstFresnel combiner (e.g., Fresnel combiner 940 includes prism 942-2optically coupled with surface 902-1 in FIG. 9D).

In some embodiments, the head-mounted display device further includesone or more prisms optically coupled with the fourth surface of thefirst Fresnel combiner (e.g., Fresnel combiner 940 includes prism 942-1optically coupled with surface 922-1 in FIG. 9D).

In some embodiments, the head-mounted display device further includes aneye tracker (e.g., eye tracker 408 in FIG. 4A) configured to determine aposition of a pupil of an eye of the wearer (e.g., pupil 406), and abeam steerer (e.g., beam steerer 404 in FIG. 4A) configured to change adirection of the light from the first light projector based on theposition of the pupil.

In some embodiments, the first light projector is configured to projectlight of a first color. In such embodiments, the head-mounted displaydevice further includes a second light projector configured to projectlight of a second color for rendering images based at least on theaugmented reality contents the second color being distinct from thefirst color (e.g., light projector 402 includes one or more lightsources, and projects ray 910-2 with first color and ray 910-4 withsecond color in FIG. 9E), and a second Fresnel combiner configured tocombine the light from the second light projector and light from theoutside of the head-mounted display device for providing an overlap ofthe rendered image and the real image (e.g., Fresnel combiner 902combines ray 910-4 and light 416 from the outside of the head-mounteddisplay device in FIG. 9E).

In some embodiments, the head-mounted display device further includes athird light projector configured to project light of a third color forrendering images based at least on the augmented reality contents. Thethird color is distinct from the first color and the second color. Thehead-mounted display device also includes a third Fresnel combinerconfigured to combine the light from the third light projector and lightfrom the outside of the head-mounted display device for providing anoverlap of the rendered image and the real image.

In accordance with some embodiments, a method providing augmentedreality contents to a wearer using a head-mounted display device thatincludes a first light projector (e.g., light projector 402 in FIG. 9A)and a first Fresnel combiner (e.g., Fresnel combiner 900) includesprojecting, with the first light projector, light for rendering an imagebased at least on the augmented reality contents (e.g., rays 910-1,910-2, and 910-3) and combining, with the first Fresnel combiner, thelight from the first light projector and light from an outside of thehead-mounted display device (e.g., light 416) for providing an overlapof the rendered image and a real image that corresponds to the lightfrom the outside of the head-mounted display device.

In accordance with some embodiments, a head-mounted display device forproviding augmented reality contents to a wearer includes a lightprojector and a pancake combiner (e.g., light projector 402 and pancakecombiner 1000 in FIG. 10A). The light projector is configured to projecta light having a first polarization for rendering images based at leaston the augmented reality contents (e.g., ray 1008-1 projected by lightprojector 402 is polarized). The pancake combiner is configured tocombine the light from the light projector (e.g., ray 1008-1) and lightfrom an outside of the head-mounted display device (e.g., light 416) forproviding an overlap of the rendered image and a real image thatcorresponds to the light from the outside of the head-mounted displaydevice. The pancake combiner is also configured to direct the light fromthe light projector toward a pupil of an eye the wearer (e.g., pupil406).

In some embodiments, the light projector (e.g., light source 402 in FIG.10A) includes a light source coupled with a polarizer. In someembodiments, the light projector includes a light source emittingpolarized light (e.g., a light source including one or more LEDs).

In some embodiments, the pancake combiner includes a first partialreflector, a first polarizer, and a second partial reflector (e.g.,pancake combiner 1000 includes partial reflector 1002, polarizer 1004,and partial reflector 1006 in FIG. 4A).

In some embodiments, the first polarizer is configured to receive thelight having the first polarization and convert the light having thefirst polarization to a light having a second polarization (e.g.,polarizer 1004 receives ray 1008-1 with a linear polarization in thez-direction and converts it to ray 1008-1 with a right-handed circularpolarization in FIG. 10A) and to receive a light having a thirdpolarization and convert the light having the third polarization to alight having a fourth polarization (e.g., polarizer 1004 receives ray1008-1 with a left-handed circular polarization and converts it to ray1008-1 with a linear polarization in the x-direction 1). The firstpartial reflector is configured to transmit at least a portion of thelight having the first polarization and to reflect at least a portion ofthe light having the fourth polarization (e.g., partial reflector 1002transmits ray 1008-1 with the linear polarization in the z-direction andreflects ray 1008-1 with the linear polarization in the x-direction).The second partial reflector is configured to reflect at least a portionof the light having the second polarization and to reflect at least aportion of the light having the third polarization (e.g., partialreflector 1006 reflects ray 1008-1 with the right-handed circularpolarization and ray 1008-1 with the left-handed circular polarization).

In some embodiments, the first polarizer is a quarter-wave plate (e.g.,polarizer 1004 is a quarter-wave plate in FIG. 10A).

In some embodiments, the first partial reflector is configured totransmit a portion of a light directed to the first partial reflectorand to reflect a portion of the light directed to the first partialreflector independent of a polarization of the light directed to thefirst partial reflector (e.g., partial reflector 1002 is a 50/50 mirrorin FIG. 10A). The second partial reflector is configured to transmit aportion of a light directed to the second partial reflector and toreflect a portion of the light directed to the second partial reflectorindependent of a polarization of the light directed to the secondpartial reflector (e.g., partial reflector 1006 is a 50/50 mirror).

In some embodiments, the first partial reflector is a first polarizationdependent mirror configured to transmit the light having the firstpolarization and to reflect the light having the fourth polarization(e.g., partial reflector 1002 is a polarization dependent mirrortransmitting light with a linear polarization in the z-direction andreflecting light with a linear polarization in the x-direction in FIG.10A). The second partial reflector is a second polarization dependentmirror configured to reflect the light having the second polarizationand to transmit light having the first polarization (e.g., partialreflector 1006 is a polarization dependent mirror reflecting light witha right-handed circular polarization and transmitting light with alinear polarization).

In some embodiments, the light having the first polarization has a firstwavelength, and the second partial reflector is a wavelength dependentreflector configured to reflect the light having the first polarizationhaving the first wavelength and to transmit light having a secondwavelength that is distinct from the first wavelength (e.g., ray 1008-1has a first wavelength, and partial reflector 1006 is awavelength-selective mirror reflecting ray 1008-1 with the firstwavelength and transmitting light with other wavelengths in FIG. 10A).

In some embodiments, the light having the first polarization has a firstwavelength, a second wavelength distinct from the first wavelength, anda third wavelength distinct from the first wavelength and the secondwavelength (e.g., ray 1008-1 includes light with three distinctwavelengths, such as blue, green and red in FIG. 10A). The secondpartial reflector is a wavelength dependent reflector configured toreflect the light having the first polarization at the first wavelength,the second wavelength, and the third wavelength, and to transmit lighthaving a fourth wavelength that is distinct from the first wavelength,the second wavelength, and the third wavelength (e.g., partial mirror1006 is a wavelength-selective mirror reflecting light with the threedistinct wavelengths of ray 1008-1 and transmitting light with otherwavelengths).

In some embodiments, the first polarization is a linear polarizationoriented in a first direction (e.g., a linear polarization in thez-direction in FIG. 10A), the second polarization is a first circularpolarization (e.g., a right-handed circular polarization), the thirdpolarization is a second circular polarization that is distinct from thefirst circular polarization (e.g., a left-handed circular polarization),and the fourth polarization is a linear polarization oriented in asecond direction that is distinct from the first direction (e.g., alinear polarization in the x-direction).

In some embodiments, the second partial reflector is curved (e.g.,partial reflector 1006 is curved in FIG. 10A).

In some embodiments, the second partial reflector includes a parabolicreflective surface (e.g., partial reflector 1006 includes a parabolicreflective surface in FIG. 10A).

In some embodiments, the second partial reflector is configured tochange a direction of the light projected by the light projector basedon the position of the pupil of the eye of the wearer by modifying aposition and/or an orientation of the parabolic partial mirror (e.g., inFIG. 10B, position of pancake combiner 1000 is modified to change thedirection of ray 1008-2 so that ray 1008-2 is directed toward pupil 406at a second gaze direction).

In some embodiments, the pancake combiner is configured to allow atleast a portion of the light from the outside of the head-mounteddisplay device transmit through the pancake combiner (e.g., pancakecombiner 1000 allows at least a portion of light 416 transmit through inFIG. 10A).

In some embodiments, the light projector is positioned away from a pathof the light, from the outside of the head-mounted display device,transmitted through the pancake combiner (e.g., light projector 402 ispositioned away from a path of light 416 in FIG. 10A).

In some embodiments, the light projector is positioned away from anoptical axis of the pancake combiner (e.g., light projector 402 ispositioned away from an optical axis of pancake combiner 1000 in FIG.10A).

In some embodiments, a beam steerer configured to change a direction ofthe light from the light projector based on the position of the pupil ofthe eye of the wearer (e.g., beam steerer 404 is configured to change adirection of light 414-1 based on position of pupil 406 in FIG. 4A).

In accordance with some embodiments, a method providing augmentedreality contents to a wearer using a head-mounted display deviceincluding a light projector and a pancake combiner (e.g., lightprojector 402 and pancake combiner 1000 in FIG. 10A) includesprojecting, with the light projector, a light having a firstpolarization (e.g., ray 1008-1) for rendering an image based at least onthe augmented reality contents and combining, with the pancake combiner,the light from the light projector (e.g., ray 1008-1) and light from anoutside of the head-mounted display device (e.g., light 416) forproviding an overlap of the rendered image and a real image thatcorresponds to the light from the outside of the head-mounted displaydevice. The pancake combiner is configured to direct the light from thelight projector toward a pupil of an eye the wearer (e.g., pupil 406).

In some embodiments, the pancake combiner includes a first partialreflector, a first polarizer, and a second partial reflector (e.g.,partial reflector 1002, polarizer 1004, and partial reflector 1006 inFIG. 10B). The method further includes transmitting through the firstpartial reflector at least a portion of the light having the firstpolarization, transmitting through the first polarizer at a first timethe light having the first polarization from the first partialreflector, and reflecting, with the second partial reflector, at least aportion of the light transmitted through the first polarizer for thefirst time toward the first polarizer. The method also includestransmitting through the first polarizer at a second time the lightreflected by the second partial reflector, and reflecting, with thefirst partial reflector, at least a portion of the light transmittedthrough the first polarizer at the second time toward the firstpolarizer. The method also includes transmitting through the firstpolarizer at a third time the light reflected by the first partialreflector, reflecting, with the second partial reflector, at least aportion of the light transmitted through the first polarizer at thethird time, transmitting through the first polarizer at a fourth timethe light reflected by the second partial reflector, and transmitting,through the first partial reflector, at least a portion of the lighttransmitted through the first polarizer at the fourth time toward thepupil of the eye of the wearer.

In accordance with some embodiments, a head-mounted display device(e.g., display device 400 in FIG. 4A) for providing augmented realitycontents to a wearer includes a light projector (e.g., light projector402) configured to project light for rendering images based at least onthe augmented reality contents, an eye tracker (e.g., eye tracker 408)configured to determine a position of a pupil of an eye of the wearer,and a beam steerer (e.g., beam steerer 404) configured to change adirection of the light from the light projector based on the position ofthe pupil.

In some embodiments, the eye tracker is integrated with the lightprojector to form an integrated combination of the eye tracker and thelight projector, and the beam steerer is optically coupled with theintegrated combination of the eye tracker and the light projector. Forexample, component 442 includes eye tracker 408 integrated with lightprojector 402 in FIG. 4D. Component 442 is optically coupled with beamsteerer 404.

In some embodiments, the device further includes one or more lensesoptically coupled with the integrated combination of the eye tracker andthe light projector (e.g., one or more lenses 412 are optically coupledwith component 442 in FIG. 4D). The light projected by the lightprojector and the light reflected from the eye of the wearer is detectedby the eye tracker are both transmitted by the one or more lenses (e.g.,light 414-1 and ray 444-1 are both transmitted by one or more lenses412).

In some embodiments, the one or more lenses are adjustable focus lenses(e.g., one or more lenses 412 are adjustable focus lenses).

In some embodiments, the device further includes a combiner configuredto combine the light from the light projector and light from an outsideof the head-mounted display device for providing an overlap of therendered image and a real image that corresponds to the light from theoutside of the head-mounted display device (e.g., combiner 410 combineslight 414-1 projected from light projector 402 and ray 418 from anoutside of display device 400 in FIG. 4A).

In some embodiments, the combiner is optically coupled with theintegrated combination of the eye tracker and the light projector (e.g.,combiner 410 is optically coupled with component 442 in FIG. 4D).

In some embodiments, the light projector is configured to project lightover an area that is within 6 mm×6 mm square area on a plane of thepupil of the eye of the wearer (e.g., light projector 402 projectslight, such as light 414-1, over an area that is within 6 mm×6 mm squarearea on reference plane 407-1 of pupil 406 in FIG. 4A), and thedirection of the light from the light projector is adjusted by the beamsteerer (e.g., beam steerer 404 adjusts the direction of light 414-1),based on the position of the pupil determined by the eye tracker.

In some embodiments, the eye tracker includes a camera configured toobtain an image of the pupil of the eye of the wearer (e.g., eye tracker408-1 includes a camera that captures an image of eye 1100 in FIG. 11A).

In some embodiments, the light projector includes an infrared lightsource configured to project infrared light toward at least a portion ofa retina of the eye of the wearer, and the eye tracker includes aninfrared detector configured to detect reflection of the infrared lightfrom the retina of the eye of the wearer. In some embodiments, theinfrared detector is a photodiode. For example, eye tracker 408-3includes an IR light source that projects ray 1114 toward the retina ofeye 1100 and eye tracker 408-3 includes an IR detector (e.g., aphotodiode) that detects ray 1116 reflected on the retina of eye 1100 inFIG. 11D.

In some embodiments, the eye tracker includes a sensor configured todetermine a distance to a surface on the eye of the wearer fordetermination of the position of the pupil of the eye of the wearer(e.g., eye tracker 408-4 includes a depth sensor, which determines adistance, such as distance 1116-1, to surface of eye 1100 fordetermination of the position of pupil 406 in FIG. 11E).

In some embodiments, the sensor is configured to determine the positionof the pupil of the eye of the wearer by scanning a predefined area thatincludes the pupil of the eye of the wearer and obtaining a contourprofile of the surface of the eye of the wearer. For example, in FIG.11E, a predefined area including pupil 406 of eye 1100 is scanned byrays having distances 1116-1, 1116-2, and 1116-3 for obtaining a contourprofile of the surface of eye 1100. As another example, in FIG. 11F, apredefined area including pupil 406 of eye 1100 is scanned by one ray ina pattern for obtaining a contour profile of the surface of eye 1100. Insome embodiments, scanning the predefined area that includes the pupilof the eye of the wearer comprises scanning in a raster pattern. In someembodiments, the contour profile of the surface of the eye of the wearercomprises a low resolution profile of the surface of the eye of thewearer.

In some embodiments, the light projector is configured to project apatterned light to the eye of the wearer, and the eye tracker is acamera configured to capture an image of a reflection of the patternedlight, reflected on a surface of the eye of the wearer. For example, eyetracker 408-2 includes a light projector that projects ray 1104-1 with astructured pattern (e.g., patterns in FIG. 11C) to eye 1100 and eyetracker 408-2 includes a camera that captures an image of ray 1104-2reflected from the surface of eye 1100 in FIG. 11B.

In some embodiments, the light projector is configured to provide alight to the eye of the wearer (e.g., eye tracker 408-2 includes a lightprojector for providing rays 1104-1 and 1106-2 to eye 1100 in FIG. 11B),and produce a reflection on a surface the eye of the wearer. At least aportion of the light produces a first reflection on a peripheral surfaceof a cornea of the eye of the wearer (e.g., ray 1104-2) and at least aportion of the light produces a second reflection on a surface of a lensof the eye of the wearer (e.g., ray 1106-2). The eye tracker isconfigured detect a first intensity of the first reflection and a secondintensity of the second reflection, and to determine, based on the firstintensity and the second intensity, the position of the pupil of the eyeof the wearer (e.g., pupil 406 of eye 1100).

In accordance with some embodiments, a method for providing augmentedreality contents to a wearer using a head-mounted display device (e.g.,display device 400 in FIG. 4A) that includes an eye tracker (e.g., eyetracker 408), a light projector (e.g., light projector 402), and a beamsteerer (e.g., beam steerer 404) includes determining, with the eyetracker, a position of a pupil of an eye of the wearer (e.g., pupil 406)and projecting, with the light projector, light (e.g., light 414-1) forrendering images based at least on the augmented reality contents. Themethod also includes changing, with the beam steerer, a direction of thelight (e.g., direction of light 414-1) from the light projector based onthe position of the pupil.

In some embodiments, determining, with the eye tracker, the position ofthe pupil of the eye of the wearer includes obtaining an image of thepupil of the eye of the wearer. For example, eye tracker 408-1 capturesan image of eye 1100 to determine position of pupil 406 in FIG. 11A).

In some embodiments, determining, with the eye tracker, the position ofthe pupil of the eye of the wearer includes providing an infrared light(e.g., ray 1114 in FIG. 11D) to the eye of the wearer, and detecting, byan infrared detector, a light reflected from a retina of the eye of thewearer (e.g., ray 1116 in FIG. 11D).

In some embodiments, determining, with the eye tracker, the position ofthe pupil of the eye of the wearer includes determining a distance to asurface on the eye of the wearer (e.g., FIG. 11E).

In some embodiments, determining the distance to the surface on the eyeof the wearer includes scanning, with a sensor, a predefined area thatincludes the pupil of the eye of the wearer and obtaining a contourprofile of the surface of the eye of the wearer (e.g., FIG. 11E and FIG.11F).

In some embodiments, determining, with the eye tracker, the position ofthe pupil of the eye of the wearer includes projecting, with the lightprojector, a patterned light to the eye of the wearer, and capturing,with a camera, an image of a reflection of the patterned light,reflected on the surface of the eye of the wearer (e.g., FIG. 11B andFIG. 11C).

In some embodiments, determining, with the eye tracker, the position ofthe pupil of the eye of the wearer includes providing, with the lightprojector, a light to the eye of the wearer and producing a reflectionon a surface the eye of the wearer. At least a portion of the lightproduces a first reflection on a peripheral surface of a cornea of theeye of the wearer and at least a portion of the light produces a secondreflection on a surface of a lens of the eye of the wearer. Determining,with the eye tracker, the position of the pupil of the eye furtherincludes detecting, with the eye tracker, a first intensity of the firstreflection and a second intensity of the second reflection, anddetermining, based on the first intensity and the second intensity, theposition of the pupil of the eye of the wearer (e.g., FIG. 11B).

In accordance with some embodiments, a method for providing augmentedreality contents to a wearer using a head-mounted display device thatincludes an eye tracking sensor, a light projector, a beam steerer, anda combiner, includes determining, with the eye tracking sensor, aposition of a pupil of an eye of the wearer and projecting, with thelight projector, light for rendering images based at least on theaugmented reality contents. The method also includes changing, with thebeam steerer, a direction of the light from the light projector based onthe position of the pupil. The light from the beam steerer is directedtoward the combiner, and the light from the beam steerer and light froman outside of the head-mounted display device are combined, by thecombiner, to provide an overlap of a rendered image and a real imagethat corresponds to the light from the outside of the head-mounteddisplay device. For example, display device 400 includes eye tracker408, light projector 402, beam steerer 404 and combiner 410 in FIG. 4A.Eye tracker 408 is configured to determine a position of pupil 406.Light projector is configured to project light 414-1 for renderingimages based at least on the augmented reality contents and beam steerer404 is configured to change a direction of light 414-1 based on theposition of pupil 406. Beam steerer directs light 414-1 to combiner 410,which reflects light 414-1 toward pupil 406 and combines light 414-1with light 416 coming from the outside display device 400.

In some embodiments, the eye tracking sensor includes a light sourceconfigured for providing an eye tracking light and a detector configuredfor receiving the eye tracking light that has been reflected from theeye. For example, eye tracker 408-5 includes light source 1121 anddetector 1123 in FIG. 11G. Determining, with the eye tracking sensor,the position of the pupil of the eye of the wearer includes providingthe eye tracking light from the light source toward the eye (e.g., ray1120 emitted by light source 1121 toward eye 1100 in FIG. 11G) anddetecting, with the detector, the eye tracking light that has beenreflected from the eye (e.g., reflected ray 1122 is detected by detector1123 in FIG. 11G). Determining the position of the pupil of the eye ofthe wearer also includes determining distance information from the eyetracking sensor to the eye based on timing information represented bythe eye tracking light from the light source and the detected eyetracking light, and determining the position of the pupil of the eye ofthe wearer based on the distance information. For example, FIG. 11Iillustrates time information for pulsed light 1120-1 emitted by lightsource 1121 at time T0, and pulses 1122-1 and 1122-2 detected bydetector 1123 at times T1 and T2, respectively. Position of pupil 406 inFIG. 11G and FIG. 11H is determined based respective optical distancestraveled by the pulses during their time-of-flights T1-T0 and T2-T1.

In some embodiments, determining the timing information based on a firsttime that corresponds to a time when the eye tracking light is providedfrom the light source toward the eye (e.g., time T1 in FIG. 11I) and asecond time that corresponds to a time when the eye tracking light,reflected from the eye, is detected by the detector (e.g., time T1 inFIG. 11I).

In some embodiments, determining the second time with an avalanchephotodiode (e.g., detector 1123 is an avalanche photodiode in FIG. 11G).

In some embodiments, the eye tracking sensor includes a light sourceconfigured for providing an eye tracking light (e.g., light source 1124in the inset of FIG. 11J) and a detector configured for receiving atleast the eye tracking light that has been reflected from the eye (e.g.,detector 1126 in the inset of FIG. 11J). Determining, with the eyetracking sensor, the position of the pupil of the eye of the wearerincludes providing the eye tracking light from the light source towardthe eye (e.g., ray 1128 in FIG. 11J) and determining the position of thepupil of the eye of the wearer based on phase information represented bythe eye tracking light from the light source and the eye tracking lightreflected from the eye (e.g., a position of pupil 406 is determinedbased on interference between ray 1130 reflected by a surface of eye1100 and reference ray 1132-3 reflected by mirror 1134 in the inset ofFIG. 11J).

In some embodiments, determining the phase information represented bythe eye tracking light from the light source and the eye tracking lightreflected from the eye includes determining a phase difference betweenthe eye tracking light from the light source and the eye tracking lightreflected from the eye (e.g., a position of pupil 406 is determinedbased on a phase difference between ray 1130 reflected by a surface ofeye 1100 and reference ray 1132-3 reflected by mirror 1134 in the insetof FIG. 11J).

In some embodiments, the eye tracking sensor is a low-resolution imagesensor or a single-pixel sensor (e.g., eye trackers 408-5 and 408-6include a low-resolution image sensor or a single-pixel sensor in FIG.11G and FIG. 11J, respectively).

In some embodiments, the eye tracking sensor includes a light sourceconfigured for providing an eye tracking light and a detector configuredfor receiving the eye tracking light that has been reflected from theeye (e.g., eye tracker 408-3 includes a light source providing ray 1114toward eye 1100 and a detector detecting ray 1116 that has beenreflected by a surface of eye 1100 in FIG. 11D). Determining, with theeye tracking sensor, the position of the pupil of the eye of the wearerincludes providing the eye tracking light from the light source towardthe eye, detecting, with the detector, the eye tracking light that hasbeen reflected from the eye, and determining the position of the pupilof the eye of the wearer based on a polarization difference between theeye tracking light from the light source and the eye tracking light thathas been reflected from the eye. For example, in FIG. 11D, polarizer1115 is positioned on the optical path of ray 1114 or on the opticalpath of reflected ray 1116, so that light detected by detector of eyetracker 408-3 is polarized. When ray 1114 enters pupil 406 and isreflected by the retina of eye 1100, eye tracker 408-3 detects ray 1116as an image with a distinguishable pattern due to birefringenceproperties of the retina of eye 1100.

In some embodiments, the eye tracking light provided toward the eye ispolarized and the method includes transmitting the eye tracking lightthat has been reflected from the eye through one or more polarizingelements (e.g., ray 1116 is polarized by polarizer 1115 in FIG. 11D).

In some embodiments, the method includes transmitting the eye trackinglight from the light source through one or more polarizing elements(e.g., ray 1114 is polarized by polarizer 1115).

In accordance with some embodiments, a head-mounted display device forproviding augmented reality contents to a wearer, the device includes aneye tracking sensor configured to determine a position of a pupil of aneye of the wearer, a light projector configured to project light forrendering images based at least on the augmented reality contents, abeam steerer configured to change a direction of the light from thelight projector based on the position of the pupil, and a combinerconfigured to combine the light from the light projector and light froman outside of the head-mounted display device for providing an overlapof the rendered image and a real image that corresponds to the lightfrom the outside of the head-mounted display device (e.g., displaydevice 400 in FIG. 4A).

In some embodiments, the eye tracking sensor includes a light sourceconfigured to provide the eye tracking light toward the eye and adetector configured to receive the eye tracking light that has beenreflected from the eye (e.g., FIG. 11G). The device is configured todetermine distance information from the eye tracking sensor to the eyebased on timing information represented by the eye tracking light fromthe light source and the detected eye tracking light. The device is alsoconfigured to determine the position of the pupil of the eye of thewearer based on the distance information.

In some embodiments, the device is configured to determine the timinginformation based on a first time that corresponds to a time when theeye tracking light is provided from the light source toward the eye anda second time that corresponds to a time when the eye tracking light,reflected from the eye, is detected by the detector (e.g., FIG. 11I).

In some embodiments, the eye tracking sensor includes an avalanchephotodiode configured to determine the second time (e.g., detector 1123is an avalanche photodiode in FIG. 11G).

In some embodiments, the eye tracking sensor includes a light sourceconfigured to provide the eye tracking light toward the eye and adetector configured to receive the eye tracking light that has beenreflected from the eye (e.g., the inset of FIG. 11J). The device isconfigured to determine the position of the pupil of the eye of thewearer based on phase information represented by the eye tracking lightfrom the light source and the eye tracking light reflected from the eye.

In some embodiments, the device is configured to determine the phaseinformation represented by the eye tracking light from the light sourceand the eye tracking light reflected from the eye includes determining aphase difference between the eye tracking light from the light sourceand the eye tracking light reflected from the eye (e.g., the inset ofFIG. 11J).

In some embodiments, the eye tracking sensor is a low-resolution imagesensor or a single-pixel sensor (e.g., eye trackers 408-5 and 408-6 inFIG. 11G and FIG. 11J, respectively).

In some embodiments, the eye tracking sensor includes a light sourceconfigured to provide an eye tracking light toward the eye and adetector configured to receive the eye tracking light that has beenreflected from the eye. The device is configured to determine theposition of the pupil of the eye of the wearer based on a polarizationdifference between the eye tracking light from the light source and theeye tracking light that has been reflected from the eye (e.g., FIG.11D).

In some embodiments, the device includes one or more polarizing elementsconfigured for transmitting the eye tracking light that has beenreflected from the eye. The eye tracking light provided toward the eyeis polarized (e.g., polarizer 1115 in FIG. 11D).

In some embodiments, the device includes one or more polarizing elementsconfigured for transmitting the eye tracking light from the light source(e.g., polarizer 1115 in FIG. 11D).

In accordance with some embodiments, a head-mounted display device forproviding images to a wearer includes a focus-supporting light projector(e.g., a combination of light projector 402 and lens 412 coupled toactuator 502, as shown in FIG. 12A) configured to project light forrendering images based at least on virtual reality contents and/oraugmented reality contents. The light projected from thefocus-supporting light projector corresponds to an image plane that isselected based at least in part on a position of a pupil of an eye ofthe wearer. The device also includes a beam steerer configured to changea path of the light projected from the focus-supporting light projectorbased on the position of the pupil of the eye of the wearer.

In some embodiments, the focus-supporting light projector includes: alight source (e.g., light source 402 in FIG. 12A) configured to projectan image; one or more lenses (e.g., one or more lenses in FIG. 12A)optically coupled with the light source to transmit the image projectedfrom the light source; and one or more actuators (e.g., actuator 502 inFIG. 12A) mechanically coupled with the one or more lenses andconfigured to move the image plane that corresponds to the lightprojected from the focus-supporting light projector.

In some embodiments, the focus-supporting light projector includes: alight source configured to project an image; and one or more lensesoptically coupled with the light source to transmit the image projectedfrom the light source, the one or more lenses including a multi-focallens (e.g., a high-speed tunable lens configured to provide differentfocal lengths for different regions of a display).

In some embodiments, the focus-supporting light projector includes: alight source configured to project an image; one or more spatial lightmodulators (e.g., spatial modulator 1202 in FIG. 12C) optically coupledwith the light source to modify the image projected from the lightsource; and one or more lenses optically coupled with the one or morespatial light modulators to transmit the image modified by the one ormore spatial light modulators.

In some embodiments, the focus-supporting light projector includes: alight source configured to project an image; one or more lensesoptically coupled with the light source to transmit the image projectedfrom the light source; and one or more actuators (e.g., actuator 502 inFIG. 12B) mechanically coupled with the light source and configured tomove the light source so that the image plane that corresponds to thelight projected from the focus-supporting light projector is moved.

In some embodiments, the focus-supporting light projector includes alens selected from a group consisting of an electro-wetting lens, aliquid lens, and a liquid crystal lens.

In some embodiments, the focus-supporting light projector includes: alight source configured to project an image; a blocker defining anaperture (e.g., blocker 1206 in FIG. 12E), the blocker optically coupledwith the light source to transmit the image projected by the lightsource through the aperture; and one or more focusing elementsconfigured to focus the image transmitted through the aperture.

In some embodiments, the device includes an eye tracker (e.g., eyetracker 408 in FIG. 12A) configured to determine the position of thepupil of the eye of the wearer.

In some embodiments, the device includes a first eye tracker (e.g., eyetracker 408-1 in FIG. 12F) configured to determine a position of a pupilof a first eye of the wearer; a second eye tracker (e.g., eye tracker408-2 in FIG. 12F) configured to determine a position of a pupil of asecond eye, of the wearer, that is distinct from the first eye of thewearer; and one or more processors configured to determine a vergencebased on the position of the pupil of the first eye of the wearer andthe position of the pupil of the second eye of the wearer. The lightprojected from the focus-supporting light projector corresponds to animage plane that is selected based on the position of the pupil of theeye of the wearer and the determined vergence (e.g., an image of object1270 corresponds to an image plane selected based on the determinedvergence).

In some embodiments, the one or more processors are configured to selectthe image plane based on the position of the pupil of the eye of thewearer and the determined vergence (e.g., the one or more processorsidentify an object the eyes are gazing at, and select an image planethat corresponds to the object the eyes are gazing at).

In accordance with some embodiments, a method for providing images to awearer is performed using a head-mounted display device that includes afocus-supporting light projector and a beam steerer. The method includesprojecting, with the focus-supporting light projector, light forrendering images based at least on virtual reality contents and/oraugmented reality contents. The light projected from thefocus-supporting light projector corresponds to an image plane that isselected based at least in part on a position of a pupil of an eye ofthe wearer. The method also includes changing, with the beam steerer, apath of the light projected from the focus-supporting light projectorbased on the position of the pupil of the eye of the wearer.

In some embodiments, the focus-supporting light projector includes: alight source configured to project an image; one or more lensesoptically coupled with the light source to transmit the image projectedfrom the light source; and one or more actuators mechanically coupledwith the one or more lenses and configured to move the image plane thatcorresponds to the light projected from the focus-supporting lightprojector. The method includes causing the one or more actuators to movethe image plane to a first location at a first time and causing the oneor more actuators to move the image plane to a second location, that isdistinct from the first location, at a second time that is distinct fromthe first time.

In some embodiments, the focus-supporting light projector includes: alight source configured to project an image; and one or more lensesoptically coupled with the light source to transmit the image projectedfrom the light source, the one or more lenses including a multi-focallens. The method includes causing the multi-focal lens to move the imageplane to a first location at a first time and causing the multi-focallens to move the image plane to a second location, that is distinct fromthe first location, at a second time that is distinct from the firsttime.

In some embodiments, the focus-supporting light projector includes: alight source configured to project an image; one or more spatial lightmodulators optically coupled with the light source to modify the imageprojected from the light source; and one or more lenses opticallycoupled with the one or more spatial light modulators to transmit theimage modified by the one or more spatial light modulators. The methodincludes causing the one or more spatial light modulators to move theimage plane to a first location at a first time and causing the one ormore spatial light modulators to move the image plane to a secondlocation, that is distinct from the first location, at a second timethat is distinct from the first time.

In some embodiments, the focus-supporting light projector includes: alight source configured to project an image; one or more lensesoptically coupled with the light source to transmit the image projectedfrom the light source; and one or more actuators mechanically coupledwith the light source and configured to move the light source so thatthe image plane that corresponds to the light projected from thefocus-supporting light projector is moved. The method includes causingthe one or more actuators to move the image plane to a first location ata first time and causing the one or more spatial light modulators tomove the image plane to a second location, that is distinct from thefirst location, at a second time that is distinct from the first time.

In some embodiments, the focus-supporting light projector includes atunable lens selected from a group consisting of an electro-wettinglens, a liquid lens, and a liquid crystal lens. The method includescausing the tunable lens to move the image plane to a first location ata first time and causing the tunable lens to move the image plane to asecond location, that is distinct from the first location, at a secondtime that is distinct from the first time.

In some embodiments, the focus-supporting light projector includes: alight source configured to project an image; a blocker defining anaperture, the blocker optically coupled with the light source totransmit the image projected by the light source through the aperture;and one or more focusing elements configured to focus the imagetransmitted through the aperture.

In some embodiments, the method includes determining, with an eyetracker, the position of the pupil of the eye of the wearer.

In some embodiments, the method includes: determining, with a first eyetracker, a position of a pupil of a first eye of the wearer;determining, with a second eye tracker, a position of a pupil of asecond eye, of the wearer, that is distinct from the first eye of thewearer; and determining, with one or more processors, a vergence basedon the position of the pupil of the first eye of the wearer and theposition of the pupil of the second eye of the wearer. The lightprojected from the focus-supporting light projector corresponds to animage plane that is selected based on the position of the pupil of theeye of the wearer and the determined vergence.

In some embodiments, the method includes selecting, with the one or moreprocessors, the image plane based on the position of the pupil of theeye of the wearer and the determined vergence.

Although various drawings illustrate operations of particular componentsor particular groups of components with respect to one eye, a personhaving ordinary skill in the art would understand that analogousoperations can be performed with respect to the other eye or both eyes.For brevity, such details are not repeated herein.

Although some of various drawings illustrate a number of logical stagesin a particular order, stages which are not order dependent may bereordered and other stages may be combined or broken out. While somereordering or other groupings are specifically mentioned, others will beapparent to those of ordinary skill in the art, so the ordering andgroupings presented herein are not an exhaustive list of alternatives.Moreover, it should be recognized that the stages could be implementedin hardware, firmware, software or any combination thereof.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the scope of the claims to the precise forms disclosed. Manymodifications and variations are possible in view of the aboveteachings. The embodiments were chosen in order to best explain theprinciples underlying the claims and their practical applications, tothereby enable others skilled in the art to best use the embodimentswith various modifications as are suited to the particular usescontemplated.

What is claimed is:
 1. A head-mounted display device for providingaugmented reality contents to a wearer, the device comprising: a lightprojector configured to project light for rendering images based atleast on the augmented reality contents; an eye tracker configured todetermine a position of a pupil of an eye of the wearer; and a beamsteerer configured to change a direction of the light from the lightprojector based on the position of the pupil.
 2. The device of claim 1,wherein: the eye tracker is integrated with the light projector to forman integrated combination of the eye tracker and the light projector;and the beam steerer is optically coupled with the integratedcombination of the eye tracker and the light projector.
 3. The device ofclaim 2, further comprising: one or more lenses optically coupled withthe integrated combination of the eye tracker and the light projector,wherein the light projected by the light projector and the lightreflected from the eye of the wearer detected by the eye tracker areboth transmitted by the one or more lenses.
 4. The device of claim 3,wherein: the one or more lenses are adjustable focus lenses.
 5. Thedevice of claim 2, further comprising: a combiner configured to combinethe light from the light projector and light from an outside of thehead-mounted display device for providing an overlap of the renderedimages and a real image that corresponds to the light from the outsideof the head-mounted display device.
 6. The device of claim 5, wherein:the combiner is optically coupled with the integrated combination of theeye tracker and the light projector.
 7. The device of claim 1, wherein:the light projector is configured to project light over an area that iswithin 6 mm×6 mm square area on a plane of the pupil of the eye of thewearer; and the direction of the light from the light projector isadjusted by the beam steerer, based on the position of the pupildetermined by the eye tracker.
 8. The device of claim 1, wherein: theeye tracker includes a camera configured to obtain an image of the pupilof the eye of the wearer.
 9. The device of claim 1, wherein: the lightprojector includes an infrared light source configured to projectinfrared light toward at least a portion of a retina of the eye of thewearer; and the eye tracker includes an infrared detector configured todetect reflection of the infrared light from the retina of the eye ofthe wearer.
 10. The device of claim 1, wherein: the eye tracker includesa sensor configured to determine a distance to a surface on the eye ofthe wearer for determination of the position of the pupil of the eye ofthe wearer.
 11. The device of claim 10, wherein: the sensor isconfigured to determine the position of the pupil of the eye of thewearer by scanning a predefined area that includes the pupil of the eyeof the wearer and obtaining a contour profile of the surface of the eyeof the wearer.
 12. The device of claim 1, wherein: the light projectoris configured to project a patterned light to the eye of the wearer; andthe eye tracker is a camera configured to capture an image of areflection of the patterned light, reflected on a surface of the eye ofthe wearer.
 13. The device of claim 1, wherein: the light projector isconfigured to: provide a light to the eye of the wearer; and produce areflection on a surface the eye of the wearer, wherein at least aportion of the light produces a first reflection on a peripheral surfaceof a cornea of the eye of the wearer and at least a portion of the lightproduces a second reflection on a surface of a lens of the eye of thewearer; and the eye tracker is configured detect a first intensity ofthe first reflection and a second intensity of the second reflection,and to determine, based on the first intensity and the second intensity,the position of the pupil of the eye of the wearer.
 14. A method forproviding augmented reality contents to a wearer using a head-mounteddisplay device that includes an eye tracker, a light projector, and abeam steerer, the method comprising: determining, with the eye tracker,a position of a pupil of an eye of the wearer; projecting, with thelight projector, light for rendering images based at least on theaugmented reality contents; and changing, with the beam steerer, adirection of the light from the light projector based on the position ofthe pupil.
 15. The method of claim 14, wherein: determining, with theeye tracker, the position of the pupil of the eye of the wearer includesobtaining an image of the pupil of the eye of the wearer.
 16. The methodof claim 14, wherein: determining, with the eye tracker, the position ofthe pupil of the eye of the wearer includes providing an infrared lightto the eye of the wearer, and detecting, by an infrared detector, alight reflected from a retina of the eye of the wearer.
 17. The methodof claim 14, wherein: determining, with the eye tracker, the position ofthe pupil of the eye of the wearer includes determining a distance to asurface on the eye of the wearer.
 18. The method of claim 17, wherein:determining the distance to the surface on the eye of the wearerincludes scanning, with a sensor, a predefined area that includes thepupil of the eye of the wearer and obtaining a contour profile of thesurface of the eye of the wearer.
 19. The method of claim 14, wherein:determining, with the eye tracker, the position of the pupil of the eyeof the wearer includes: projecting, with the light projector, apatterned light to the eye of the wearer; and capturing, with a camera,an image of a reflection of the patterned light, reflected on a surfaceof the eye of the wearer.
 20. The method of claim 14, wherein:determining, with the eye tracker, the position of the pupil of the eyeof the wearer includes: providing, with the light projector, a light tothe eye of the wearer; producing a reflection on a surface the eye ofthe wearer, wherein at least a portion of the light produces a firstreflection on a peripheral surface of a cornea of the eye of the wearerand at least a portion of the light produces a second reflection on asurface of a lens of the eye of the wearer; and detecting, with the eyetracker, a first intensity of the first reflection and a secondintensity of the second reflection; and determining, based on the firstintensity and the second intensity, the position of the pupil of the eyeof the wearer.